Merge lp:~barry/ubuntuone-storage-protocol/lp1077092 into lp:ubuntuone-storage-protocol

Proposed by Barry Warsaw
Status: Merged
Approved by: dobey
Approved revision: 161
Merged at revision: 157
Proposed branch: lp:~barry/ubuntuone-storage-protocol/lp1077092
Merge into: lp:ubuntuone-storage-protocol
Diff against target: 151 lines (+56/-38)
2 files modified
tests/test_client.py (+37/-21)
ubuntuone/storageprotocol/client.py (+19/-17)
To merge this branch: bzr merge lp:~barry/ubuntuone-storage-protocol/lp1077092
Reviewer Review Type Date Requested Status
dobey (community) Abstain
Roberto Alsina (community) Approve
Alejandro J. Cura (community) Approve
Review via email: mp+146528@code.launchpad.net

Commit message

Switch from python-oauth to python-oauthlib for using OAuth.

Description of the change

Use the supported and Python 3 compatible python-oauthlib instead of the abandoned and no longer supported oauth library.

To post a comment you must log in.
Revision history for this message
Alejandro J. Cura (alecu) wrote :

Thanks for working on this!

One comment: the oauth_authenticate method changed changed here is only used (afaict) from lp:ubuntuone-client ActionQueue.authenticate(), and with a different signature. That means that with this change a new branch for ubuntuone-client would be needed.

Instead, I propose that the changes to the signature of oauth_authenticate in this branch are reverted, and that the compatibility properties defined in lp:~barry/ubuntuone-client/lp1077089 are used instead, like:

135 + client = Client(token.key, token.secret, consumer.key, consumer.secret ...

Does this seem reasonable?

review: Needs Fixing
Revision history for this message
Barry Warsaw (barry) wrote :

On Feb 06, 2013, at 02:04 PM, Alejandro J. Cura wrote:

>One comment: the oauth_authenticate method changed changed here is only used
>(afaict) from lp:ubuntuone-client ActionQueue.authenticate(), and with a
>different signature. That means that with this change a new branch for
>ubuntuone-client would be needed.

Ah right, my changes to ubuntuone-client didn't change the call signature over
there.

>Instead, I propose that the changes to the signature of oauth_authenticate in
>this branch are reverted, and that the compatibility properties defined in
>lp:~barry/ubuntuone-client/lp1077089 are used instead, like:
>
>135 + client = Client(token.key, token.secret, consumer.key, consumer.secret ...
>
>Does this seem reasonable?

It does, good catch.

I suppose if we wanted to get rid of this API backward compatibility hack, we
could do that in a future branch.

Branch update.

Revision history for this message
Alejandro J. Cura (alecu) wrote :

I'm trying to run both branches together like this:
(make sure to ./run-tests in the u1sp branch first, so protobuf files get generated).

alecu@bollo:~/canonical/ubuntuone-client/review_lp1077089$ u1sdtool -q; PYTHONPATH=.:~/canonical/ubuntuone-storage-protocol/review_lp1077092/ ./bin/ubuntuone-syncdaemon --debug

The syncdaemon debug log shows that the authentication fails when using the new branches, and I'm able to authenticate using the system's syncdaemon, so there must be something that's working differently.

review: Needs Fixing
Revision history for this message
Barry Warsaw (barry) wrote :

Phew! That was a fun one to track down and fix. ;)

Let's start by disabling the dbus auto-restart of the ubuntuone-syncdaemon on
a production system. You have to do this or the `u1sdtool -q` command is
ineffectual. It does quit the daemon, but dbus re-activates it so the one in
bin/ubuntuone-syncdaemon will never be able to start.

The easiest way I found for doing this was to comment out the Name and Exec
lines in /usr/share/dbus-1/services/com.ubuntuone.SyncDaemon.service, then
kill -HUP dbus to re-read its service files. Now when you `u1sdtool -q`, it
kills the running syncdaemon and dbus won't restart it.

On Feb 07, 2013, at 12:13 AM, Alejandro J. Cura wrote:

>I'm trying to run both branches together like this:
>(make sure to ./run-tests in the u1sp branch first, so protobuf files get generated).
>
>alecu@bollo:~/canonical/ubuntuone-client/review_lp1077089$ u1sdtool -q; PYTHONPATH=.:~/canonical/ubuntuone-storage-protocol/review_lp1077092/ ./bin/ubuntuone-syncdaemon --debug
>
>The syncdaemon debug log shows that the authentication fails when using the
>new branches, and I'm able to authenticate using the system's syncdaemon, so
>there must be something that's working differently.

Yep. It turns out that I was confused by the confusing differences in
nomenclature. python-oauth uses the older terms "consumer" and "access token"
while python-oauthlib uses the RFC 5849 terms "client" and "resource owner".
When doing the conversion between the two regimes, I got them backwards and
was passing the first four arguments to oauthlib's Client constructor in the
reverse order.

This should now be fixed. I've updated the LP: #1077092 merge proposal branch
for the correct order in client.py, and it looks to me like both the tests
pass, and the cross-branch test you describe above successfully connects.

Hopefully this resolves any outstanding issues with the two branches, but
please let me know if it's still not working!

Revision history for this message
Alejandro J. Cura (alecu) wrote :

> Let's start by disabling the dbus auto-restart of the ubuntuone-syncdaemon on
> a production system. You have to do this or the `u1sdtool -q` command is
> ineffectual. It does quit the daemon, but dbus re-activates it so the one in
> bin/ubuntuone-syncdaemon will never be able to start.
>
> The easiest way I found for doing this was to comment out the Name and Exec
> lines in /usr/share/dbus-1/services/com.ubuntuone.SyncDaemon.service, then
> kill -HUP dbus to re-read its service files. Now when you `u1sdtool -q`, it
> kills the running syncdaemon and dbus won't restart it.

Thanks a lot for researching the workaround for this!
I'm ashamed to admit the autorestarting of syncdaemon has been a long standing bug that we've never prioritized.
Sorry again about that.

> Hopefully this resolves any outstanding issues with the two branches, but
> please let me know if it's still not working!

It's connecting perfectly now, thanks again for working on all these branches.

review: Approve
Revision history for this message
Roberto Alsina (ralsina) wrote :

works ok IRL for me

review: Approve
Revision history for this message
dobey (dobey) wrote :

Am just putting a negative vote at the moment, as we can't immediately land this, as it will break ubuntuone-client trunk, and as we need to fix the issue with getting branches landed there, due to squid (ncsa_auth) crashing on raring.

review: Needs Information
Revision history for this message
Barry Warsaw (barry) wrote :

NP. Will this (and the other branches) get auto-merged once trunk is fixed, or do you need me to do anything else?

Revision history for this message
dobey (dobey) wrote :

I'll handle getting it merged, once the squid related kinks are worked around, as it's already been approved.

Revision history for this message
dobey (dobey) :
review: Abstain
Revision history for this message
Ubuntu One Auto Pilot (otto-pilot) wrote :

The attempt to merge lp:~barry/ubuntuone-storage-protocol/lp1077092 into lp:ubuntuone-storage-protocol failed. Below is the output from the failed tests.

running build

*** Cannot find protoc; is the protobuf-compiler package installed?

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== modified file 'tests/test_client.py'
--- tests/test_client.py 2012-12-03 19:45:43 +0000
+++ tests/test_client.py 2013-02-07 21:46:21 +0000
@@ -38,6 +38,8 @@
38import unittest38import unittest
39import uuid39import uuid
4040
41from urlparse import urlparse, parse_qsl
42
41from twisted.application import internet, service43from twisted.application import internet, service
42from twisted.internet import defer44from twisted.internet import defer
43from twisted.internet.defer import Deferred, inlineCallbacks45from twisted.internet.defer import Deferred, inlineCallbacks
@@ -48,10 +50,12 @@
48from ubuntuone.storageprotocol.client import (50from ubuntuone.storageprotocol.client import (
49 StorageClient, CreateUDF, ListVolumes, DeleteVolume, GetDelta, Unlink,51 StorageClient, CreateUDF, ListVolumes, DeleteVolume, GetDelta, Unlink,
50 Authenticate, MakeFile, MakeDir, PutContent, Move, BytesMessageProducer,52 Authenticate, MakeFile, MakeDir, PutContent, Move, BytesMessageProducer,
51 oauth, TwistedTimestampChecker, tx_timestamp_checker,53 TwistedTimestampChecker, tx_timestamp_checker,
52)54)
55
53from ubuntuone.storageprotocol import volumes56from ubuntuone.storageprotocol import volumes
54from tests import test_delta_info57from tests import test_delta_info
58from oauthlib.oauth1 import Client
5559
56# let's not get picky about aatributes outside __init__ in tests60# let's not get picky about aatributes outside __init__ in tests
57# pylint: disable=W020161# pylint: disable=W0201
@@ -795,31 +799,43 @@
795799
796 def test_oauth_authenticate_uses_server_timestamp(self):800 def test_oauth_authenticate_uses_server_timestamp(self):
797 """The oauth authentication uses the server timestamp."""801 """The oauth authentication uses the server timestamp."""
798 fromcandt_call = []802 timestamps = []
799803
800 fake_token = oauth.OAuthToken('token', 'token_secret')804 original_sign = Client.sign
801 fake_consumer = oauth.OAuthConsumer('consumer_key', 'consumer_secret')805
802806 def fake_sign(self, *args, **kwargs):
803 fake_timestamp = object()807 """A fake Client.sign()."""
808 url, headers, body = original_sign(self, *args, **kwargs)
809 timestamps.extend(
810 value for name, value in parse_qsl(urlparse(url).query)
811 if name == 'oauth_timestamp')
812 return url, headers, body
813
814 fake_timestamp = '801'
804 timestamp_d = Deferred()815 timestamp_d = Deferred()
805 self.patch(tx_timestamp_checker, "get_faithful_time",816 self.patch(tx_timestamp_checker, "get_faithful_time",
806 lambda: timestamp_d)817 lambda: timestamp_d)
807 original_fromcandt = oauth.OAuthRequest.from_consumer_and_token818
808819 self.patch(Client, 'sign', fake_sign)
809 @staticmethod
810 def fake_from_consumer_and_token(**kwargs):
811 """A fake from_consumer_and_token."""
812 fromcandt_call.append(kwargs)
813 return original_fromcandt(**kwargs)
814
815 self.patch(oauth.OAuthRequest, "from_consumer_and_token",
816 fake_from_consumer_and_token)
817 protocol = FakedProtocol()820 protocol = FakedProtocol()
818 protocol.oauth_authenticate(fake_consumer, fake_token)821 # For backward compatibility of the API, oauth_authenticate() takes a
819 self.assertEqual(len(fromcandt_call), 0)822 # consumer object and a token object. Both of those must have .key
823 # and .secret attributes. In modern OAuth1 (i.e. RFC 5849)
824 # terminology, the consumer is really the OAuth client, and the token
825 # is really the OAuth resource owner.
826
827 class OAuthClient:
828 key = 'consumer_key'
829 secret = 'consumer_secret'
830
831 class OAuthResourceOwner:
832 key = 'token'
833 secret = 'token_secret'
834 protocol.oauth_authenticate(OAuthResourceOwner, OAuthClient)
835 self.assertEqual(len(timestamps), 0)
820 timestamp_d.callback(fake_timestamp)836 timestamp_d.callback(fake_timestamp)
821 parameters = fromcandt_call[0]["parameters"]837 self.assertEqual(len(timestamps), 1)
822 self.assertEqual(parameters["oauth_timestamp"], fake_timestamp)838 self.assertEqual(timestamps[0], fake_timestamp)
823839
824840
825class RootResource(resource.Resource):841class RootResource(resource.Resource):
826842
=== modified file 'ubuntuone/storageprotocol/client.py'
--- ubuntuone/storageprotocol/client.py 2012-12-03 19:45:43 +0000
+++ ubuntuone/storageprotocol/client.py 2013-02-07 21:46:21 +0000
@@ -39,7 +39,8 @@
3939
40from functools import partial40from functools import partial
41from itertools import chain41from itertools import chain
42from oauth import oauth42from urlparse import urlparse, parse_qsl
43from oauthlib.oauth1 import Client, SIGNATURE_PLAINTEXT, SIGNATURE_TYPE_QUERY
4344
44from twisted.internet.protocol import ClientFactory45from twisted.internet.protocol import ClientFactory
45from twisted.internet import reactor, defer46from twisted.internet import reactor, defer
@@ -143,29 +144,30 @@
143 def oauth_authenticate(self, consumer, token, metadata=None):144 def oauth_authenticate(self, consumer, token, metadata=None):
144 """Authenticate to a server using the OAuth provider.145 """Authenticate to a server using the OAuth provider.
145146
146 @param consumer: the OAuth consumer to authenticate with.147 @param consumer: the OAuth consumer (a.k.a. client in RFC 5849) to
147 @type consumer: `oauth.OAuthConsumer`148 authenticate with.
148 @param token: a previously acquired OAuth access token.149 @type consumer: object having both a `.key` and `.secret` attribute.
149 @type consumer: `oauth.OAuthToken`150 @param token: a previously acquired OAuth access token (a.k.a.
150 @param kwargs: key/values to send as metadata151 resource owner in RFC 5849).
152 @type token: object having both a `.key` and `.secret` attribute.
153 @param metadata: key/values to send as metadata
151154
152 Return a deferred that will get called with the request155 Return a deferred that will get called with the request
153 object when completed.156 object when completed.
154157
155 """158 """
156 timestamp = yield tx_timestamp_checker.get_faithful_time()159 timestamp = yield tx_timestamp_checker.get_faithful_time()
157 req = oauth.OAuthRequest.from_consumer_and_token(160 client = Client(consumer.key, consumer.secret,
158 oauth_consumer=consumer,161 token.key, token.secret,
159 token=token,162 signature_method=SIGNATURE_PLAINTEXT,
160 parameters={"oauth_timestamp": timestamp},163 signature_type=SIGNATURE_TYPE_QUERY,
161 http_method="GET",164 timestamp=timestamp)
162 http_url="storage://server")165 url, headers, body = client.sign('storage://server')
163 req.sign_request(166 # Parse out the authentication parameters from the query string.
164 oauth.OAuthSignatureMethod_PLAINTEXT(), consumer, token)
165
166 # Make sure all parameter values are strings.
167 auth_parameters = dict(167 auth_parameters = dict(
168 (key, str(value)) for key, value in req.parameters.iteritems())168 (name, value) for name, value in
169 parse_qsl(urlparse(url).query)
170 if name.startswith('oauth_'))
169 p = Authenticate(self, auth_parameters, metadata=metadata)171 p = Authenticate(self, auth_parameters, metadata=metadata)
170 p.start()172 p.start()
171 result = yield p.deferred173 result = yield p.deferred

Subscribers

People subscribed via source and target branches