Gwibber

Merge lp:~robru/gwibber/facebook into lp:~barry/gwibber/py3

  1. facebook
  2. Merge into py3
Proposed by Robert Bruce Park
Status: Merged
Merged at revision: 1452
Proposed branch: lp:~robru/gwibber/facebook
Merge into: lp:~barry/gwibber/py3
Diff against target: 2732 lines (+805/-1301)
22 files modified
gwibber/gwibber/protocols/facebook.py (+197/-2)
gwibber/gwibber/protocols/foursquare.py (+2/-2)
gwibber/gwibber/protocols/statusnet.py (+0/-26)
gwibber/gwibber/protocols/twitter.py (+11/-14)
gwibber/gwibber/testing/mocks.py (+60/-0)
gwibber/gwibber/tests/data/facebook-full.dat (+1/-0)
gwibber/gwibber/tests/data/facebook-login.dat (+1/-0)
gwibber/gwibber/tests/test_avatars.py (+13/-12)
gwibber/gwibber/tests/test_dbus.py (+3/-0)
gwibber/gwibber/tests/test_download.py (+21/-18)
gwibber/gwibber/tests/test_facebook.py (+304/-0)
gwibber/gwibber/tests/test_flickr.py (+19/-18)
gwibber/gwibber/tests/test_foursquare.py (+5/-4)
gwibber/gwibber/tests/test_model.py (+1/-1)
gwibber/gwibber/tests/test_protocols.py (+3/-4)
gwibber/gwibber/tests/test_shortener.py (+6/-5)
gwibber/gwibber/tests/test_twitter.py (+77/-81)
gwibber/gwibber/utils/base.py (+1/-1)
gwibber/gwibber/utils/download.py (+73/-42)
gwibber/gwibber/utils/model.py (+7/-2)
gwibber/microblog/plugins/facebook/__init__.py (+0/-480)
gwibber/microblog/plugins/statusnet/__init__.py (+0/-589)
To merge this branch: bzr merge lp:~robru/gwibber/facebook
Reviewer Review Type Date Requested Status
Ken VanDine behold libsoup! Pending
Barry Warsaw Pending
Review via email: mp+129336@code.launchpad.net

Description of the change

Port the facebook plugin from old gwibber to new gwibber, including the prerequisite port from urllib to libsoup, which was necessary in order to achieve proxy support and HTTP DELETE requests, which Facebook uses.

To post a comment you must log in.
Revision history for this message
Robert Bruce Park (robru) wrote :

Oh barry, when you're reviewing this, keep an eye out for the changes in Downloader._download. libsoup doesn't raise exceptions like urllib does, so I had originally written some code to just log the errors, but after writing that I came across some tests you'd written that test for the exceptions being raised. I didn't really understand how to change those tests to check the log for the logged error instead of expecting the exception to be raised, so I commented out the logging code and replaced it with a manual raising of HTTPError. If you could take a look at that, delete the 'raise HTTPError' bit, uncomment the logging bit, and fix the tests to accept the logging behavior instead of the raising behavior, that would be *super*.

If you disagree and insist on the raising HTTPError behavior, then the code is fine as-is.

Thanks!

Revision history for this message
Barry Warsaw (barry) wrote :

The branch looks great, thanks for taking on these two big tasks!

I did just a little bit of reformatting for lines-too-long and whitespace
cleanup. The really nice thing is that I was also able to remove the FakeData
mock, since FakeSoupMessage replaces that everywhere.

Question: why did you remove this test?

=== modified file 'gwibber/gwibber/tests/test_download.py'
--- gwibber/gwibber/tests/test_download.py 2012-10-05 01:19:29 +0000
+++ gwibber/gwibber/tests/test_download.py 2012-10-12 14:39:18 +0000
> @@ -232,9 +238,6 @@
> Downloader('http://localhost:9180/auth',
> username='bob', password='wrong').get_string()
> self.assertEqual(cm.exception.code, 401)
> - # Read the error body.
> - message = cm.exception.read().decode('utf-8')
> - self.assertEqual(message, 'username/password mismatch')

Is it because libsoup doesn't give you access to the exception message?

On Oct 12, 2012, at 01:40 AM, Robert Bruce Park wrote:

>Oh barry, when you're reviewing this, keep an eye out for the changes in
>Downloader._download. libsoup doesn't raise exceptions like urllib does, so I
>had originally written some code to just log the errors, but after writing
>that I came across some tests you'd written that test for the exceptions
>being raised. I didn't really understand how to change those tests to check
>the log for the logged error instead of expecting the exception to be raised,
>so I commented out the logging code and replaced it with a manual raising of
>HTTPError. If you could take a look at that, delete the 'raise HTTPError'
>bit, uncomment the logging bit, and fix the tests to accept the logging
>behavior instead of the raising behavior, that would be *super*.
>
>If you disagree and insist on the raising HTTPError behavior, then the code
>is fine as-is.

I do like raising an exception in this case because it stops further
processing, although in effect it only short-circuits the rate limiting (still
useful though). Note that it wouldn't *have* to be an HTTPError, it could be
a gwibber-service specific exception, but it's probably not worth it. I added
a comment to that code.

This is just looking so fantastic, great work!

I think we can finally kill off the entire microblog directory now. :)

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'gwibber/gwibber/protocols/facebook.py'
2--- gwibber/gwibber/protocols/facebook.py 2012-09-19 22:21:39 +0000
3+++ gwibber/gwibber/protocols/facebook.py 2012-10-12 01:37:25 +0000
4@@ -19,8 +19,203 @@
5 ]
6
7
8-from gwibber.utils.base import Base
9+import logging
10+
11+from datetime import datetime, timedelta
12+
13+from gwibber.utils.base import Base, feature
14+from gwibber.utils.download import get_json
15+from gwibber.utils.time import parsetime, iso8601utc
16+
17+
18+# 'id' can be the id of *any* facebook object
19+# https://developers.facebook.com/docs/reference/api/
20+URL_BASE = 'https://{subdomain}.facebook.com/'
21+PERMALINK = URL_BASE.format(subdomain='www') + '{id}'
22+API_BASE = URL_BASE.format(subdomain='graph') + '{id}'
23+ME_URL = API_BASE.format(id='me')
24+LIMIT = 100
25+
26+
27+log = logging.getLogger('gwibber.service')
28
29
30 class Facebook(Base):
31- pass
32+ def _whoami(self, authdata):
33+ """Identify the authenticating user."""
34+ me_data = get_json(
35+ ME_URL, dict(access_token=self._account.access_token))
36+ self._account.user_id = me_data.get('id')
37+ self._account.user_name = me_data.get('name')
38+
39+ def _is_error(self, data):
40+ """Is the return data an error response?"""
41+ error = data.get('error')
42+ if error is None:
43+ return False
44+ log.error('Facebook error ({} {}): {}'.format(
45+ error.get('code'), error.get('type'), error.get('message')))
46+ return True
47+
48+ def _publish_entry(self, entry):
49+ message_id = entry.get('id')
50+ if message_id is None:
51+ # We can't do much with this entry.
52+ return
53+
54+ args = dict(
55+ stream='messages',
56+ message=entry.get('message', ''),
57+ url=PERMALINK.format(id=message_id),
58+ icon_uri=entry.get('icon', ''),
59+ link_picture=entry.get('picture', ''),
60+ link_name=entry.get('name', ''),
61+ link_url=entry.get('link', ''),
62+ link_desc=entry.get('description', ''),
63+ link_caption=entry.get('caption', ''),
64+ )
65+ from_record = entry.get('from')
66+ if from_record is not None:
67+ args['sender'] = sender_id = from_record.get('id', '')
68+ args['sender_nick'] = from_record.get('name', '')
69+ args['from_me'] = (sender_id == self._account.user_id)
70+ # Normalize the timestamp.
71+ timestamp = entry.get('updated_time', entry.get('created_time'))
72+ if timestamp is not None:
73+ args['timestamp'] = iso8601utc(parsetime(timestamp))
74+ like_count = entry.get('likes', {}).get('count')
75+ if like_count is not None:
76+ args['likes'] = like_count
77+ args['liked'] = (like_count > 0)
78+ # Parse comments now.
79+ all_comments = []
80+ for comment_data in entry.get('comments', {}).get('data', []):
81+ comment_message = comment_data.get('message')
82+ if comment_message is not None:
83+ all_comments.append(comment_message)
84+ args['comments'] = all_comments
85+ self._publish(message_id, **args)
86+
87+ @feature
88+ def receive(self, since=None):
89+ """Retrieve a list of Facebook objects.
90+
91+ A maximum of 100 objects are requested.
92+
93+ :param since: Only get objects posted since this date. If not given,
94+ then only objects younger than 10 days are retrieved. The value
95+ is a number seconds since the epoch.
96+ :type since: float
97+ """
98+ access_token = self._get_access_token()
99+ if since is None:
100+ when = datetime.now() - timedelta(days=10)
101+ else:
102+ when = datetime.fromtimestamp(since)
103+ entries = []
104+ url = ME_URL + '/home'
105+ params = dict(access_token=access_token,
106+ since=when.isoformat(),
107+ limit=LIMIT)
108+ # Now access Facebook and follow pagination until we have at least
109+ # LIMIT number of entries, or we've reached the end of pages.
110+ while True:
111+ response = get_json(url, params)
112+ if self._is_error(response):
113+ # We'll just use what we have so far, if anything.
114+ break
115+ data = response.get('data')
116+ if data is None:
117+ # I guess we're done.
118+ break
119+ entries.extend(data)
120+ if len(entries) >= LIMIT:
121+ break
122+ # We haven't gotten the requested number of entries. Follow the
123+ # next page if there is one to try to get more.
124+ pages = response.get('paging')
125+ if pages is None:
126+ break
127+ # The 'next' key has the full link to follow; no additional
128+ # parameters are needed. Specifically, this link will already
129+ # include the access_token, and any since/limit values.
130+ url = pages.get('next')
131+ params = None
132+ if url is None:
133+ # I guess there are no more next pages.
134+ break
135+ # We've gotten everything Facebook is going to give us. Now, decipher
136+ # the data and publish it.
137+ # https://developers.facebook.com/docs/reference/api/post/
138+ for entry in entries:
139+ self._publish_entry(entry)
140+
141+ def _like(self, obj_id, method):
142+ url = API_BASE.format(id=obj_id) + '/likes'
143+ token = self._get_access_token()
144+
145+ if not get_json(url, method=method, params=dict(access_token=token)):
146+ log.error('Failed to {} like {} on Facebook'.format(method, obj_id))
147+
148+ @feature
149+ def like(self, obj_id):
150+ """Like any arbitrary object on Facebook.
151+
152+ This includes messages, statuses, wall posts, events, etc.
153+ """
154+ self._like(obj_id, 'POST')
155+
156+ @feature
157+ def unlike(self, obj_id):
158+ """Unlike any arbitrary object on Facebook.
159+
160+ This includes messages, statuses, wall posts, events, etc.
161+ """
162+ self._like(obj_id, 'DELETE')
163+
164+ def _send(self, obj_id, message, endpoint):
165+ url = API_BASE.format(id=obj_id) + endpoint
166+ token = self._get_access_token()
167+
168+ result = get_json(
169+ url,
170+ method='POST',
171+ params=dict(access_token=token, message=message))
172+ new_id = result.get('id')
173+ if new_id is None:
174+ log.error('Failed sending to Facebook: {!r}'.format(result))
175+ return
176+
177+ url = API_BASE.format(id=new_id)
178+ entry = get_json(url, params=dict(access_token=token))
179+ self._publish_entry(entry)
180+
181+ @feature
182+ def send(self, message, obj_id='me'):
183+ """Write a message on somebody or something's wall.
184+
185+ If you don't specify an obj_id, it defaults to your wall.
186+ obj_id can be any type of Facebook object that has a wall, be
187+ it a user, an app, a company, an event, etc.
188+ """
189+ self._send(obj_id, message, '/feed')
190+
191+ @feature
192+ def send_thread(self, obj_id, message):
193+ """Write a comment on some existing status message.
194+
195+ obj_id can be the id of any Facebook object that supports
196+ being commented on, which will generally be Posts.
197+ """
198+ self._send(obj_id, message, '/comments')
199+
200+ @feature
201+ def delete(self, obj_id):
202+ """Delete any Facebook object that you are the owner of."""
203+ url = API_BASE.format(id=obj_id)
204+ token = self._get_access_token()
205+
206+ if not get_json(url, method='DELETE', params=dict(access_token=token)):
207+ log.error('Failed to delete {} on Facebook'.format(method, obj_id))
208+ else:
209+ self._unpublish(obj_id)
210
211=== modified file 'gwibber/gwibber/protocols/foursquare.py'
212--- gwibber/gwibber/protocols/foursquare.py 2012-10-10 14:33:01 +0000
213+++ gwibber/gwibber/protocols/foursquare.py 2012-10-12 01:37:25 +0000
214@@ -57,8 +57,8 @@
215 class FourSquare(Base):
216 def _whoami(self, authdata):
217 """Identify the authenticating user."""
218- data = get_json(SELF_URL.format(
219- access_token=self._account.access_token))
220+ data = get_json(
221+ SELF_URL.format(access_token=self._account.access_token))
222 user = data.get('response', {}).get('user', {})
223 self._account.secret_token = authdata.get('TokenSecret')
224 self._account.user_name = _full_name(user)
225
226=== removed file 'gwibber/gwibber/protocols/statusnet.py'
227--- gwibber/gwibber/protocols/statusnet.py 2012-09-19 22:21:39 +0000
228+++ gwibber/gwibber/protocols/statusnet.py 1970-01-01 00:00:00 +0000
229@@ -1,26 +0,0 @@
230-# Copyright (C) 2012 Canonical Ltd
231-#
232-# This program is free software: you can redistribute it and/or modify
233-# it under the terms of the GNU General Public License version 2 as
234-# published by the Free Software Foundation.
235-#
236-# This program is distributed in the hope that it will be useful,
237-# but WITHOUT ANY WARRANTY; without even the implied warranty of
238-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
239-# GNU General Public License for more details.
240-#
241-# You should have received a copy of the GNU General Public License
242-# along with this program. If not, see <http://www.gnu.org/licenses/>.
243-
244-"""The Status.net protocol plugin."""
245-
246-__all__ = [
247- 'StatusNet',
248- ]
249-
250-
251-from gwibber.utils.base import Base
252-
253-
254-class StatusNet(Base):
255- pass
256
257=== modified file 'gwibber/gwibber/protocols/twitter.py'
258--- gwibber/gwibber/protocols/twitter.py 2012-10-10 17:26:01 +0000
259+++ gwibber/gwibber/protocols/twitter.py 2012-10-12 01:37:25 +0000
260@@ -26,7 +26,7 @@
261
262 from oauthlib.oauth1 import Client
263 from urllib.error import HTTPError
264-from urllib.parse import quote, urlsplit, urlunsplit
265+from urllib.parse import quote
266
267 from gwibber.utils.base import Base, feature
268 from gwibber.utils.download import RateLimiter as BaseRateLimiter, get_json
269@@ -73,6 +73,7 @@
270 def _get_url(self, url, data=None):
271 """Access the Twitter API with correct OAuth signed headers."""
272 do_post = data is not None
273+ method = 'POST' if do_post else 'GET'
274
275 # "Client" == "Consumer" in oauthlib parlance.
276 client_key = self._account.auth.parameters['ConsumerKey']
277@@ -91,10 +92,9 @@
278 # All we care about is the headers, which will contain the
279 # Authorization header necessary to satisfy OAuth.
280 uri, headers, body = oauth_client.sign(
281- url, body=data, headers=headers,
282- http_method='POST' if do_post else 'GET')
283+ url, body=data, headers=headers, http_method=method)
284
285- return get_json(url, params=data, headers=headers, post=do_post,
286+ return get_json(url, params=data, headers=headers, method=method,
287 rate_limiter=self._rate_limiter)
288
289 def _publish_tweet(self, tweet):
290@@ -299,21 +299,18 @@
291 def __init__(self):
292 self._limits = {}
293
294- def _sanitize_url(self, url):
295+ def _sanitize_url(self, uri):
296 # Cache the URL sans any query parameters.
297- parts = list(urlsplit(url))
298- # Set the query element to the empty string.
299- parts[3] = ''
300- return urlunsplit(parts)
301+ return uri.host + uri.path
302
303- def wait(self, request):
304+ def wait(self, message):
305 # If we haven't seen this URL, default to no wait.
306- seconds = self._limits.get(self._sanitize_url(request.full_url), 0)
307+ seconds = self._limits.get(self._sanitize_url(message.get_uri()), 0)
308 time.sleep(seconds)
309
310- def update(self, response):
311- info = response.info()
312- url = self._sanitize_url(response.geturl())
313+ def update(self, message):
314+ info = message.response_headers
315+ url = self._sanitize_url(message.get_uri())
316 # This is time in the future, in UTC epoch seconds, at which the
317 # current rate limiting window expires.
318 rate_reset = info.get('X-Rate-Limit-Reset')
319
320=== modified file 'gwibber/gwibber/testing/mocks.py'
321--- gwibber/gwibber/testing/mocks.py 2012-10-05 17:58:23 +0000
322+++ gwibber/gwibber/testing/mocks.py 2012-10-12 01:37:25 +0000
323@@ -15,6 +15,7 @@
324 """Mocks, doubles, and fakes for testing."""
325
326 __all__ = [
327+ 'FakeSoupMessage',
328 'FakeData',
329 'FakeOpen',
330 'LogMock',
331@@ -30,6 +31,7 @@
332 from logging.handlers import QueueHandler
333 from pkg_resources import resource_listdir, resource_string
334 from queue import Empty, Queue
335+from urllib.parse import urlsplit
336
337 from gwibber.utils.logging import LOG_FORMAT
338
339@@ -43,6 +45,64 @@
340 NEWLINE = '\n'
341
342
343+class FakeSoupMessage:
344+ """Mimic a Soup.Message that returns canned data."""
345+
346+ def __init__(self, path, resource, charset='utf-8', headers=None,
347+ response_code=200):
348+ # resource_string() always returns bytes.
349+ self._data = resource_string(path, resource)
350+ self.call_count = 0
351+ self._charset = charset
352+ self._headers = {} if headers is None else headers
353+ self.status_code = response_code
354+
355+ @property
356+ def response_body(self):
357+ return self
358+
359+ @property
360+ def response_headers(self):
361+ return self
362+
363+ @property
364+ def request_headers(self):
365+ return self
366+
367+ def flatten(self):
368+ return self
369+
370+ def get_data(self):
371+ return self._data
372+
373+ def get_as_bytes(self):
374+ return self._data
375+
376+ def get_content_type(self):
377+ return 'application/x-mock-data', dict(charset=self._charset)
378+
379+ def get_uri(self):
380+ pieces = urlsplit(self.url)
381+ class FakeUri:
382+ host = pieces.netloc
383+ path = pieces.path
384+ return FakeUri()
385+
386+ def get(self, header, default=None):
387+ return self._headers.get(header, default)
388+
389+ def append(self, header, value):
390+ self._headers[header] = value
391+
392+ def set_request(self, *args):
393+ return
394+
395+ def new(self, method, url):
396+ self.call_count += 1
397+ self.method = method
398+ self.url = url
399+ return self
400+
401 # A test double which shortens the given URL by returning the hex hash of the
402 # source URL.
403 class FakeOpen:
404
405=== added file 'gwibber/gwibber/tests/data/facebook-full.dat'
406--- gwibber/gwibber/tests/data/facebook-full.dat 1970-01-01 00:00:00 +0000
407+++ gwibber/gwibber/tests/data/facebook-full.dat 2012-10-12 01:37:25 +0000
408@@ -0,0 +1,1 @@
409+{"paging": {"previous": "https://graph.facebook.com/101/home?access_token=ABC&limit=25&since=1348682101&__previous=1"}, "data": [{"picture": "https://fbexternal-a.akamaihd.net/rush.jpg", "from": {"category": "Entertainment", "name": "Rush is a Band", "id": "117402931676347"}, "name": "Rush is a Band Blog", "comments": {"count": 0}, "actions": [{"link": "https://www.facebook.com/117402931676347/posts/287578798009078", "name": "Comment"}, {"link": "https://www.facebook.com/117402931676347/posts/287578798009078", "name": "Like"}], "updated_time": "2012-09-26T17:34:00+0000", "caption": "www.rushisaband.com", "link": "http://www.rushisaband.com/blog/Rush-Clockwork-Angels-tour", "likes": {"count": 16, "data": [{"name": "Alex Lifeson", "id": "801"}, {"name": "Vlada Lee", "id": "801"}, {"name": "Richard Peart", "id": "803"}, {"name": "Eric Lifeson", "id": "804"}]}, "created_time": "2012-09-26T17:34:00+0000", "message": "Rush takes off to the Great White North", "icon": "https://s-static.ak.facebook.com/rsrc.php/v2/yD/r/a.gif", "type": "link", "id": "108", "status_type": "shared_story", "description": "Rush is a Band: Neil Peart, Geddy Lee, Alex Lifeson"}, {"picture": "https://images.gibson.com/Rush_Clockwork-Angels_t.jpg", "likes": {"count": 27, "data": [{"name": "Tracy Lee", "id": "805"}, {"name": "Wendy Peart", "id": "806"}, {"name": "Vlada Lifeson", "id": "807"}, {"name": "Chevy Lee", "id": "808"}]}, "from": {"category": "Entertainment", "name": "Rush is a Band", "id": "117402931676347"}, "name": "Top 10 Alex Lifeson Guitar Moments", "comments": {"count": 5, "data": [{"created_time": "2012-09-26T17:16:00+0000", "message": "OK Don...10) Headlong Flight", "from": {"name": "Bruce Peart", "id": "809"}, "id": "117402931676347_386054134801436_3235476"}, {"created_time": "2012-09-26T17:49:06+0000", "message": "No Cygnus X-1 Bruce? I call shenanigans!", "from": {"name": "Don Lee", "id": "810"}, "id": "117402931676347_386054134801436_3235539"}]}, "actions": [{"link": "https://www.facebook.com/117402931676347/posts/386054134801436", "name": "Comment"}, {"link": "https://www.facebook.com/117402931676347/posts/386054134801436", "name": "Like"}], "updated_time": "2012-09-26T17:49:06+0000", "caption": "www2.gibson.com", "link": "http://www2.gibson.com/Alex-Lifeson.aspx", "shares": {"count": 11}, "created_time": "2012-09-26T16:42:15+0000", "message": "http://www2.gibson.com/Alex-Lifeson-0225-2011.aspx", "icon": "https://s-static.ak.facebook.com/rsrc.php/v2/yD/r/a.gif", "type": "link", "id": "109", "status_type": "shared_story", "description": "For millions of Rush fans old and new, it\u2019s a pleasure"}]}
410\ No newline at end of file
411
412=== added file 'gwibber/gwibber/tests/data/facebook-login.dat'
413--- gwibber/gwibber/tests/data/facebook-login.dat 1970-01-01 00:00:00 +0000
414+++ gwibber/gwibber/tests/data/facebook-login.dat 2012-10-12 01:37:25 +0000
415@@ -0,0 +1,1 @@
416+{"id": "801", "name": "Bart Person"}
417\ No newline at end of file
418
419=== modified file 'gwibber/gwibber/tests/test_avatars.py'
420--- gwibber/gwibber/tests/test_avatars.py 2012-09-11 20:55:59 +0000
421+++ gwibber/gwibber/tests/test_avatars.py 2012-10-12 01:37:25 +0000
422@@ -35,10 +35,11 @@
423 except ImportError:
424 import mock
425
426-from gwibber.testing.mocks import FakeData
427+from gwibber.testing.mocks import FakeSoupMessage
428 from gwibber.utils.avatar import Avatar
429
430
431+@mock.patch('gwibber.utils.download._soup', mock.Mock())
432 class TestAvatars(unittest.TestCase):
433 """Test Avatar logic."""
434
435@@ -65,8 +66,8 @@
436 # hashlib.sha1('fake_url'.encode('utf-8')).hexdigest()
437 '4f37e5dc9d38391db1728048344c3ab5ff8cecb2'])
438
439- @mock.patch('gwibber.utils.download.urlopen',
440- FakeData('gwibber.tests.data', 'ubuntu.png'))
441+ @mock.patch('gwibber.utils.download.Soup.Message',
442+ FakeSoupMessage('gwibber.tests.data', 'ubuntu.png'))
443 def test_cache_filled_on_miss(self):
444 # When the cache is empty, downloading an avatar from a given url
445 # fills the cache with the image data.
446@@ -76,17 +77,17 @@
447 # not yet exist. It is created on demand.
448 self.assertFalse(os.path.isdir(cache_dir))
449 Avatar.get_image('http://example.com')
450- # urlopen() was called once. Get the mock and check it.
451- from gwibber.utils.download import urlopen
452- self.assertEqual(urlopen.call_count, 1)
453+ # Soup.Message() was called once. Get the mock and check it.
454+ from gwibber.utils.download import Soup
455+ self.assertEqual(Soup.Message.call_count, 1)
456 # Now the file is there.
457 self.assertEqual(os.listdir(cache_dir),
458 # hashlib.sha1('http://example.com'
459 # .encode('utf-8')).hexdigest()
460 ['89dce6a446a69d6b9bdc01ac75251e4c322bcdff'])
461
462- @mock.patch('gwibber.utils.download.urlopen',
463- FakeData('gwibber.tests.data', 'ubuntu.png'))
464+ @mock.patch('gwibber.utils.download.Soup.Message',
465+ FakeSoupMessage('gwibber.tests.data', 'ubuntu.png'))
466 def test_cache_used_on_hit(self):
467 # When the cache already contains the file, it is not downloaded.
468 with mock.patch('gwibber.utils.avatar.CACHE_DIR',
469@@ -99,16 +100,16 @@
470 # Get the image, resulting in a cache hit.
471 path = Avatar.get_image('http://example.com')
472 # No download occurred. Check the mock.
473- from gwibber.utils.download import urlopen
474- self.assertEqual(urlopen.call_count, 0)
475+ from gwibber.utils.download import Soup
476+ self.assertEqual(Soup.Message.call_count, 0)
477 # Confirm that the resulting cache image is actually a PNG.
478 with open(path, 'rb') as raw:
479 # This is the PNG file format magic number, living in the first 8
480 # bytes of the file.
481 self.assertEqual(raw.read(8), bytes.fromhex('89504E470D0A1A0A'))
482
483- @mock.patch('gwibber.utils.download.urlopen',
484- FakeData('gwibber.tests.data', 'ubuntu.png'))
485+ @mock.patch('gwibber.utils.download.Soup.Message',
486+ FakeSoupMessage('gwibber.tests.data', 'ubuntu.png'))
487 def test_cache_file_contains_image(self):
488 # The image is preserved in the cache file.
489 with mock.patch('gwibber.utils.avatar.CACHE_DIR', self._avatar_cache):
490
491=== modified file 'gwibber/gwibber/tests/test_dbus.py'
492--- gwibber/gwibber/tests/test_dbus.py 2012-10-05 12:50:57 +0000
493+++ gwibber/gwibber/tests/test_dbus.py 2012-10-12 01:37:25 +0000
494@@ -111,6 +111,9 @@
495 '/com/gwibber/Service')
496 iface = dbus.Interface(obj, 'com.Gwibber.Service')
497 # TODO Add more cases as more protocols are added.
498+ self.assertEqual(json.loads(iface.GetFeatures('facebook')),
499+ ['delete', 'like', 'receive', 'send', 'send_thread',
500+ 'unlike'])
501 self.assertEqual(json.loads(iface.GetFeatures('twitter')),
502 ['delete', 'follow', 'home', 'like', 'list', 'lists',
503 'mentions', 'private', 'receive', 'retweet',
504
505=== modified file 'gwibber/gwibber/tests/test_download.py'
506--- gwibber/gwibber/tests/test_download.py 2012-10-05 01:19:29 +0000
507+++ gwibber/gwibber/tests/test_download.py 2012-10-12 01:37:25 +0000
508@@ -35,7 +35,7 @@
509 from wsgiref.simple_server import WSGIRequestHandler, make_server
510
511 from gwibber.utils.download import Downloader, get_json
512-from gwibber.testing.mocks import FakeData
513+from gwibber.testing.mocks import FakeSoupMessage
514
515 try:
516 # Python 3.3
517@@ -152,43 +152,49 @@
518 self.assertEqual(get_json('http://localhost:9180/json'),
519 dict(answer='hello'))
520
521- @mock.patch('gwibber.utils.download.urlopen',
522- FakeData('gwibber.tests.data', 'json-utf-8.dat', 'utf-8'))
523+ @mock.patch('gwibber.utils.download._soup', mock.Mock())
524+ @mock.patch('gwibber.utils.download.Soup.Message',
525+ FakeSoupMessage('gwibber.tests.data', 'json-utf-8.dat', 'utf-8'))
526 def test_json_explicit_utf_8(self):
527 # RFC 4627 $3 with explicit charset=utf-8.
528 self.assertEqual(get_json('http://example.com'),
529 dict(yes='ÑØ'))
530
531- @mock.patch('gwibber.utils.download.urlopen',
532- FakeData('gwibber.tests.data', 'json-utf-8.dat', None))
533+ @mock.patch('gwibber.utils.download._soup', mock.Mock())
534+ @mock.patch('gwibber.utils.download.Soup.Message',
535+ FakeSoupMessage('gwibber.tests.data', 'json-utf-8.dat', None))
536 def test_json_implicit_utf_8(self):
537 # RFC 4627 $3 with implicit charset=utf-8.
538 self.assertEqual(get_json('http://example.com'),
539 dict(yes='ÑØ'))
540
541- @mock.patch('gwibber.utils.download.urlopen',
542- FakeData('gwibber.tests.data', 'json-utf-16le.dat', None))
543+ @mock.patch('gwibber.utils.download._soup', mock.Mock())
544+ @mock.patch('gwibber.utils.download.Soup.Message',
545+ FakeSoupMessage('gwibber.tests.data', 'json-utf-16le.dat', None))
546 def test_json_implicit_utf_16le(self):
547 # RFC 4627 $3 with implicit charset=utf-16le.
548 self.assertEqual(get_json('http://example.com'),
549 dict(yes='ÑØ'))
550
551- @mock.patch('gwibber.utils.download.urlopen',
552- FakeData('gwibber.tests.data', 'json-utf-16be.dat', None))
553+ @mock.patch('gwibber.utils.download._soup', mock.Mock())
554+ @mock.patch('gwibber.utils.download.Soup.Message',
555+ FakeSoupMessage('gwibber.tests.data', 'json-utf-16be.dat', None))
556 def test_json_implicit_utf_16be(self):
557 # RFC 4627 $3 with implicit charset=utf-16be.
558 self.assertEqual(get_json('http://example.com'),
559 dict(yes='ÑØ'))
560
561- @mock.patch('gwibber.utils.download.urlopen',
562- FakeData('gwibber.tests.data', 'json-utf-32le.dat', None))
563+ @mock.patch('gwibber.utils.download._soup', mock.Mock())
564+ @mock.patch('gwibber.utils.download.Soup.Message',
565+ FakeSoupMessage('gwibber.tests.data', 'json-utf-32le.dat', None))
566 def test_json_implicit_utf_32le(self):
567 # RFC 4627 $3 with implicit charset=utf-32le.
568 self.assertEqual(get_json('http://example.com'),
569 dict(yes='ÑØ'))
570
571- @mock.patch('gwibber.utils.download.urlopen',
572- FakeData('gwibber.tests.data', 'json-utf-32be.dat', None))
573+ @mock.patch('gwibber.utils.download._soup', mock.Mock())
574+ @mock.patch('gwibber.utils.download.Soup.Message',
575+ FakeSoupMessage('gwibber.tests.data', 'json-utf-32be.dat', None))
576 def test_json_implicit_utf_32be(self):
577 # RFC 4627 $3 with implicit charset=utf-32be.
578 self.assertEqual(get_json('http://example.com'),
579@@ -209,14 +215,14 @@
580 # Test posting data.
581 self.assertEqual(get_json('http://localhost:9180/post',
582 params=dict(one=1, two=2, three=3),
583- post=True),
584+ method='POST'),
585 dict(one=1, two=2, three=3))
586
587 def test_params_get(self):
588 # Test getting with query string URL.
589 self.assertEqual(get_json('http://localhost:9180/post',
590 params=dict(one=1, two=2, three=3),
591- post=False),
592+ method='GET'),
593 dict(one=1, two=2, three=3))
594
595 def test_unauthorized(self):
596@@ -232,9 +238,6 @@
597 Downloader('http://localhost:9180/auth',
598 username='bob', password='wrong').get_string()
599 self.assertEqual(cm.exception.code, 401)
600- # Read the error body.
601- message = cm.exception.read().decode('utf-8')
602- self.assertEqual(message, 'username/password mismatch')
603
604 def test_authorized(self):
605 # Test a URL that requires authorization, and has the proper
606
607=== added file 'gwibber/gwibber/tests/test_facebook.py'
608--- gwibber/gwibber/tests/test_facebook.py 1970-01-01 00:00:00 +0000
609+++ gwibber/gwibber/tests/test_facebook.py 2012-10-12 01:37:25 +0000
610@@ -0,0 +1,304 @@
611+# Copyright (C) 2012 Canonical Ltd
612+#
613+# This program is free software: you can redistribute it and/or modify
614+# it under the terms of the GNU General Public License version 2 as
615+# published by the Free Software Foundation.
616+#
617+# This program is distributed in the hope that it will be useful,
618+# but WITHOUT ANY WARRANTY; without even the implied warranty of
619+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
620+# GNU General Public License for more details.
621+#
622+# You should have received a copy of the GNU General Public License
623+# along with this program. If not, see <http://www.gnu.org/licenses/>.
624+
625+"""Test the Facebook plugin."""
626+
627+__all__ = [
628+ 'TestFacebook',
629+ ]
630+
631+
632+import unittest
633+
634+from gi.repository import Dee
635+
636+from gwibber.protocols.facebook import Facebook
637+from gwibber.testing.helpers import FakeAccount
638+from gwibber.testing.mocks import FakeSoupMessage, LogMock
639+from gwibber.utils.base import Base
640+from gwibber.utils.model import COLUMN_TYPES
641+
642+
643+try:
644+ # Python 3.3
645+ from unittest import mock
646+except ImportError:
647+ import mock
648+
649+
650+# Create a test model that will not interfere with the user's environment.
651+# We'll use this object as a mock of the real model.
652+TestModel = Dee.SharedModel.new('com.Gwibber.TestSharedModel')
653+TestModel.set_schema_full(COLUMN_TYPES)
654+
655+
656+@mock.patch('gwibber.utils.download._soup', mock.Mock())
657+class TestFacebook(unittest.TestCase):
658+ """Test the Facebook API."""
659+
660+ def setUp(self):
661+ self.account = FakeAccount()
662+ self.protocol = Facebook(self.account)
663+ # Enable sub-thread synchronization, and mock out the loggers.
664+ Base._SYNCHRONIZE = True
665+ self.log_mock = LogMock('gwibber.utils.base',
666+ 'gwibber.protocols.facebook')
667+
668+ def tearDown(self):
669+ # Stop log mocking, and return sub-thread operation to asynchronous.
670+ self.log_mock.stop()
671+ Base._SYNCHRONIZE = False
672+ # Reset the database.
673+ TestModel.clear()
674+
675+ @mock.patch('gwibber.utils.authentication.Authentication.login',
676+ return_value=dict(AccessToken='abc'))
677+ @mock.patch('gwibber.utils.download.Soup.Message',
678+ FakeSoupMessage('gwibber.tests.data', 'facebook-login.dat'))
679+ def test_successful_login(self, mock):
680+ # Test that a successful response from graph.facebook.com returning
681+ # the user's data, sets up the account dict correctly.
682+ self.protocol._login()
683+ self.assertEqual(self.account.access_token, 'abc')
684+ self.assertEqual(self.account.user_name, 'Bart Person')
685+ self.assertEqual(self.account.user_id, '801')
686+
687+ @mock.patch('gwibber.utils.authentication.Authentication.login',
688+ return_value=None)
689+ def test_login_unsuccessful_authentication(self, mock):
690+ # The user is not already logged in, but the act of logging in fails.
691+ self.protocol._login()
692+ self.assertIsNone(self.account.access_token)
693+ self.assertIsNone(self.account.user_name)
694+
695+ @mock.patch('gwibber.utils.authentication.Authentication.login',
696+ return_value={})
697+ def test_unsuccessful_login_no_access_token(self, mock):
698+ # When Authentication.login() returns a dictionary, but that does not
699+ # have the AccessToken key, then the account data is not updated.
700+ self.protocol._login()
701+ self.assertIsNone(self.account.access_token)
702+ self.assertIsNone(self.account.user_name)
703+
704+ @mock.patch('gwibber.utils.authentication.Authentication.login',
705+ return_value=dict(AccessToken='abc'))
706+ @mock.patch('gwibber.protocols.facebook.get_json',
707+ return_value=dict(
708+ error=dict(message='Bad access token',
709+ type='OAuthException',
710+ code=190)))
711+ def test_error_response(self, *mocks):
712+ self.protocol('receive')
713+ self.assertEqual(self.log_mock.empty(trim=False), """\
714+Facebook error (190 OAuthException): Bad access token
715+""")
716+
717+ @mock.patch('gwibber.utils.download.Soup.Message',
718+ FakeSoupMessage('gwibber.tests.data', 'facebook-full.dat'))
719+ @mock.patch('gwibber.utils.base.Model', TestModel)
720+ @mock.patch('gwibber.protocols.facebook.Facebook._login',
721+ return_value=True)
722+ def test_receive(self, *mocks):
723+ # Receive the wall feed for a user.
724+ self.maxDiff = None
725+ self.account.access_token = 'abc'
726+ self.protocol.receive()
727+ self.assertEqual(TestModel.get_n_rows(), 2)
728+ self.assertEqual(list(TestModel.get_row(0)), [
729+ [['facebook', 'faker/than fake', '108']],
730+ 'messages',
731+ '117402931676347',
732+ 'Rush is a Band',
733+ False,
734+ '2012-09-26T17:34:00',
735+ 'Rush takes off to the Great White North',
736+ '',
737+ 'https://s-static.ak.facebook.com/rsrc.php/v2/yD/r/a.gif',
738+ 'https://www.facebook.com/108',
739+ '',
740+ '',
741+ '',
742+ '',
743+ 16.0,
744+ True,
745+ '',
746+ '',
747+ '',
748+ 'https://fbexternal-a.akamaihd.net/rush.jpg',
749+ 'Rush is a Band Blog',
750+ 'http://www.rushisaband.com/blog/Rush-Clockwork-Angels-tour',
751+ 'Rush is a Band: Neil Peart, Geddy Lee, Alex Lifeson',
752+ 'www.rushisaband.com',
753+ '',
754+ '',
755+ '',
756+ '',
757+ '',
758+ '',
759+ '',
760+ '',
761+ '',
762+ [],
763+ '',
764+ '',
765+ ''])
766+ self.assertEqual(list(TestModel.get_row(1)), [
767+ [['facebook', 'faker/than fake', '109']],
768+ 'messages',
769+ '117402931676347',
770+ 'Rush is a Band',
771+ False,
772+ '2012-09-26T17:49:06',
773+ 'http://www2.gibson.com/Alex-Lifeson-0225-2011.aspx',
774+ '',
775+ 'https://s-static.ak.facebook.com/rsrc.php/v2/yD/r/a.gif',
776+ 'https://www.facebook.com/109',
777+ '',
778+ '',
779+ '',
780+ '',
781+ 27.0,
782+ True,
783+ '',
784+ '',
785+ '',
786+ 'https://images.gibson.com/Rush_Clockwork-Angels_t.jpg',
787+ 'Top 10 Alex Lifeson Guitar Moments',
788+ 'http://www2.gibson.com/Alex-Lifeson.aspx',
789+ 'For millions of Rush fans old and new, it’s a pleasure',
790+ 'www2.gibson.com',
791+ '',
792+ '',
793+ '',
794+ '',
795+ '',
796+ '',
797+ '',
798+ '',
799+ '',
800+ ['OK Don...10) Headlong Flight',
801+ 'No Cygnus X-1 Bruce? I call shenanigans!'],
802+ '',
803+ '',
804+ ''])
805+
806+ # XXX We really need full coverage of the receive() method, including
807+ # cases where some data is missing, or can't be converted
808+ # (e.g. timestamps), and paginations.
809+
810+ @mock.patch('gwibber.protocols.facebook.get_json',
811+ return_value=dict(id='post_id'))
812+ def test_send_to_my_wall(self, get_json):
813+ token = self.protocol._get_access_token = mock.Mock(return_value='face')
814+ publish = self.protocol._publish_entry = mock.Mock()
815+
816+ self.protocol.send('I can see the writing on my wall.')
817+
818+ token.assert_called_once_with()
819+ publish.assert_called_with({'id': 'post_id'})
820+ self.assertEqual(
821+ get_json.mock_calls,
822+ [mock.call('https://graph.facebook.com/me/feed',
823+ method='POST',
824+ params=dict(access_token='face',
825+ message='I can see the writing on my wall.')),
826+ mock.call('https://graph.facebook.com/post_id',
827+ params=dict(access_token='face'))
828+ ])
829+
830+ @mock.patch('gwibber.protocols.facebook.get_json',
831+ return_value=dict(id='post_id'))
832+ def test_send_to_my_friends_wall(self, get_json):
833+ token = self.protocol._get_access_token = mock.Mock(return_value='face')
834+ publish = self.protocol._publish_entry = mock.Mock()
835+
836+ self.protocol.send('I can see the writing on my friend\'s wall.',
837+ 'friend_id')
838+
839+ token.assert_called_once_with()
840+ publish.assert_called_with({'id': 'post_id'})
841+ self.assertEqual(
842+ get_json.mock_calls,
843+ [mock.call(
844+ 'https://graph.facebook.com/friend_id/feed',
845+ method='POST',
846+ params=dict(
847+ access_token='face',
848+ message='I can see the writing on my friend\'s wall.')),
849+ mock.call('https://graph.facebook.com/post_id',
850+ params=dict(access_token='face'))
851+ ])
852+
853+ @mock.patch('gwibber.protocols.facebook.get_json',
854+ return_value=dict(id='comment_id'))
855+ def test_send_thread(self, get_json):
856+ token = self.protocol._get_access_token = mock.Mock(return_value='face')
857+ publish = self.protocol._publish_entry = mock.Mock()
858+
859+ self.protocol.send_thread('post_id', 'Some witty response!')
860+
861+ token.assert_called_once_with()
862+ publish.assert_called_with({'id': 'comment_id'})
863+ self.assertEqual(
864+ get_json.mock_calls,
865+ [mock.call(
866+ 'https://graph.facebook.com/post_id/comments',
867+ method='POST',
868+ params=dict(
869+ access_token='face',
870+ message='Some witty response!')),
871+ mock.call('https://graph.facebook.com/comment_id',
872+ params=dict(access_token='face'))
873+ ])
874+
875+ @mock.patch('gwibber.protocols.facebook.get_json',
876+ return_value=True)
877+ def test_like(self, get_json):
878+ token = self.protocol._get_access_token = mock.Mock(return_value='face')
879+
880+ self.protocol.like('post_id')
881+
882+ token.assert_called_once_with()
883+ get_json.assert_called_once_with(
884+ 'https://graph.facebook.com/post_id/likes',
885+ method='POST',
886+ params=dict(access_token='face'))
887+
888+ @mock.patch('gwibber.protocols.facebook.get_json',
889+ return_value=True)
890+ def test_unlike(self, get_json):
891+ token = self.protocol._get_access_token = mock.Mock(return_value='face')
892+
893+ self.protocol.unlike('post_id')
894+
895+ token.assert_called_once_with()
896+ get_json.assert_called_once_with(
897+ 'https://graph.facebook.com/post_id/likes',
898+ method='DELETE',
899+ params=dict(access_token='face'))
900+
901+ @mock.patch('gwibber.protocols.facebook.get_json',
902+ return_value=True)
903+ def test_delete(self, get_json):
904+ token = self.protocol._get_access_token = mock.Mock(return_value='face')
905+ unpublish = self.protocol._unpublish = mock.Mock()
906+
907+ self.protocol.delete('post_id')
908+
909+ token.assert_called_once_with()
910+ get_json.assert_called_once_with(
911+ 'https://graph.facebook.com/post_id',
912+ method='DELETE',
913+ params=dict(access_token='face'))
914+ unpublish.assert_called_once_with('post_id')
915
916=== modified file 'gwibber/gwibber/tests/test_flickr.py'
917--- gwibber/gwibber/tests/test_flickr.py 2012-10-10 14:33:01 +0000
918+++ gwibber/gwibber/tests/test_flickr.py 2012-10-12 01:37:25 +0000
919@@ -25,7 +25,7 @@
920
921 from gwibber.protocols.flickr import Flickr
922 from gwibber.testing.helpers import FakeAccount
923-from gwibber.testing.mocks import FakeData, LogMock
924+from gwibber.testing.mocks import FakeSoupMessage, LogMock
925 from gwibber.utils.base import Base
926 from gwibber.utils.model import COLUMN_INDICES, COLUMN_TYPES
927
928@@ -42,6 +42,7 @@
929 TestModel.set_schema_full(COLUMN_TYPES)
930
931
932+@mock.patch('gwibber.utils.download._soup', mock.Mock())
933 class TestFlickr(unittest.TestCase):
934 """Test the Flickr API."""
935
936@@ -77,8 +78,8 @@
937 No Flickr user id available (account: faker/than fake)
938 """)
939
940- @mock.patch('gwibber.utils.download.urlopen',
941- FakeData('gwibber.tests.data', 'flickr-nophotos.dat'))
942+ @mock.patch('gwibber.utils.download.Soup.Message',
943+ FakeSoupMessage('gwibber.tests.data', 'flickr-nophotos.dat'))
944 @mock.patch('gwibber.utils.base.Model', TestModel)
945 def test_already_logged_in(self):
946 # Try to get the data when already logged in.
947@@ -92,8 +93,8 @@
948 # But also no photos.
949 self.assertEqual(TestModel.get_n_rows(), 0)
950
951- @mock.patch('gwibber.utils.download.urlopen',
952- FakeData('gwibber.tests.data', 'flickr-nophotos.dat'))
953+ @mock.patch('gwibber.utils.download.Soup.Message',
954+ FakeSoupMessage('gwibber.tests.data', 'flickr-nophotos.dat'))
955 def test_unsuccessful_login(self):
956 # The user is not already logged in, but the act of logging in
957 # fails.
958@@ -113,8 +114,8 @@
959 No Flickr user id available (account: faker/than fake)
960 """)
961
962- @mock.patch('gwibber.utils.download.urlopen',
963- FakeData('gwibber.tests.data', 'flickr-nophotos.dat'))
964+ @mock.patch('gwibber.utils.download.Soup.Message',
965+ FakeSoupMessage('gwibber.tests.data', 'flickr-nophotos.dat'))
966 @mock.patch('gwibber.utils.base.Model', TestModel)
967 def test_successful_login(self):
968 # The user is not already logged in, but the act of logging in
969@@ -131,8 +132,8 @@
970 # But also no photos.
971 self.assertEqual(TestModel.get_n_rows(), 0)
972
973- @mock.patch('gwibber.utils.download.urlopen',
974- FakeData('gwibber.tests.data', 'flickr-nophotos.dat'))
975+ @mock.patch('gwibber.utils.download.Soup.Message',
976+ FakeSoupMessage('gwibber.tests.data', 'flickr-nophotos.dat'))
977 @mock.patch('gwibber.utils.authentication.Authentication.login',
978 # No AccessToken, so for all intents-and-purposes; fail!
979 return_value=dict(username='Bob Dobbs',
980@@ -153,8 +154,8 @@
981 No Flickr user id available (account: faker/than fake)
982 """)
983
984- @mock.patch('gwibber.utils.download.urlopen',
985- FakeData('gwibber.tests.data', 'flickr-nophotos.dat'))
986+ @mock.patch('gwibber.utils.download.Soup.Message',
987+ FakeSoupMessage('gwibber.tests.data', 'flickr-nophotos.dat'))
988 @mock.patch('gwibber.utils.authentication.Authentication.login',
989 # login() callback never happens.
990 return_value=None)
991@@ -163,7 +164,7 @@
992 # AccessToken, but this fails.
993 self.protocol('receive')
994 self.assertEqual(self.log_mock.empty(), """\
995-No Flickr authentication results received
996+No Flickr authentication results received.
997 Flickr: No NSID available
998 Gwibber operation exception:
999 Traceback (most recent call last):
1000@@ -172,8 +173,8 @@
1001 No Flickr user id available (account: faker/than fake)
1002 """)
1003
1004- @mock.patch('gwibber.utils.download.urlopen',
1005- FakeData('gwibber.tests.data', 'flickr-nophotos.dat'))
1006+ @mock.patch('gwibber.utils.download.Soup.Message',
1007+ FakeSoupMessage('gwibber.tests.data', 'flickr-nophotos.dat'))
1008 @mock.patch('gwibber.utils.authentication.Authentication.login',
1009 return_value=dict(username='Bob Dobbs',
1010 user_nsid='bob',
1011@@ -211,8 +212,8 @@
1012 method='flickr.photos.getContactsPublicPhotos',
1013 ))
1014
1015- @mock.patch('gwibber.utils.download.urlopen',
1016- FakeData('gwibber.tests.data', 'flickr-nophotos.dat'))
1017+ @mock.patch('gwibber.utils.download.Soup.Message',
1018+ FakeSoupMessage('gwibber.tests.data', 'flickr-nophotos.dat'))
1019 @mock.patch('gwibber.utils.base.Model', TestModel)
1020 def test_no_photos(self):
1021 # The JSON data in response to the GET request returned no photos.
1022@@ -221,8 +222,8 @@
1023 self.protocol('receive')
1024 self.assertEqual(TestModel.get_n_rows(), 0)
1025
1026- @mock.patch('gwibber.utils.download.urlopen',
1027- FakeData('gwibber.tests.data', 'flickr-full.dat'))
1028+ @mock.patch('gwibber.utils.download.Soup.Message',
1029+ FakeSoupMessage('gwibber.tests.data', 'flickr-full.dat'))
1030 @mock.patch('gwibber.utils.base.Model', TestModel)
1031 def test_flickr_data(self):
1032 # Test the undocumented and apparently crufty reformatting of the raw
1033
1034=== modified file 'gwibber/gwibber/tests/test_foursquare.py'
1035--- gwibber/gwibber/tests/test_foursquare.py 2012-09-25 20:25:42 +0000
1036+++ gwibber/gwibber/tests/test_foursquare.py 2012-10-12 01:37:25 +0000
1037@@ -25,7 +25,7 @@
1038
1039 from gwibber.protocols.foursquare import FourSquare
1040 from gwibber.testing.helpers import FakeAccount
1041-from gwibber.testing.mocks import FakeData, LogMock
1042+from gwibber.testing.mocks import FakeSoupMessage, LogMock
1043 from gwibber.utils.model import COLUMN_TYPES
1044
1045 try:
1046@@ -44,6 +44,7 @@
1047 # Ensure synchronicity between the main thread and the sub-thread. Also, set
1048 # up the loggers for the modules-under-test so that we can assert their error
1049 # messages.
1050+@mock.patch('gwibber.utils.download._soup', mock.Mock())
1051 @mock.patch.dict('gwibber.utils.base.__dict__', {'_SYNCHRONIZE': True})
1052 class TestFourSquare(unittest.TestCase):
1053 """Test the FourSquare API."""
1054@@ -86,8 +87,8 @@
1055 self.assertEqual(self.account.user_id, '1234567')
1056
1057 @mock.patch('gwibber.utils.base.Model', TestModel)
1058- @mock.patch('gwibber.utils.download.urlopen',
1059- FakeData('gwibber.tests.data', 'foursquare-full.dat'))
1060+ @mock.patch('gwibber.utils.download.Soup.Message',
1061+ FakeSoupMessage('gwibber.tests.data', 'foursquare-full.dat'))
1062 @mock.patch('gwibber.protocols.foursquare.FourSquare._login',
1063 return_value=True)
1064 def test_receive(self, *mocks):
1065@@ -103,7 +104,7 @@
1066 'https://api.foursquare.com/v2/checkins/50574c9ce4b0a9a6e84433a0' +
1067 '?oauth_token=tokeny goodness&v=20120917', '', '', '', '', 0.0,
1068 False, '', '', '', '', '', '', '', '', '', '', '', '', '', '',
1069- '', '', '', '', '', '', '',
1070+ '', '', '', [], '', '', '',
1071 ]
1072 for got, want in zip(TestModel.get_row(0), expected):
1073 self.assertEqual(got, want)
1074
1075=== modified file 'gwibber/gwibber/tests/test_model.py'
1076--- gwibber/gwibber/tests/test_model.py 2012-09-19 23:22:48 +0000
1077+++ gwibber/gwibber/tests/test_model.py 2012-10-12 01:37:25 +0000
1078@@ -40,4 +40,4 @@
1079 ['aas', 's', 's', 's', 'b', 's', 's', 's',
1080 's', 's', 's', 's', 's', 's', 'd', 'b', 's', 's',
1081 's', 's', 's', 's', 's', 's', 's', 's', 's', 's',
1082- 's', 's', 's', 's', 's', 's', 's', 's', 's'])
1083+ 's', 's', 's', 's', 's', 'as', 's', 's', 's'])
1084
1085=== modified file 'gwibber/gwibber/tests/test_protocols.py'
1086--- gwibber/gwibber/tests/test_protocols.py 2012-10-04 04:50:42 +0000
1087+++ gwibber/gwibber/tests/test_protocols.py 2012-10-12 01:37:25 +0000
1088@@ -207,10 +207,9 @@
1089 self.assertEqual(V('likes'), 10)
1090 self.assertTrue(V('liked'))
1091 # All the other columns have empty string values.
1092- empty_columns = set(COLUMN_NAMES) - set(['message_ids', 'stream',
1093- 'sender', 'sender_nick',
1094- 'from_me', 'timestamp',
1095- 'message', 'likes', 'liked'])
1096+ empty_columns = set(COLUMN_NAMES) - set(
1097+ ['message_ids', 'stream', 'sender', 'sender_nick', 'from_me',
1098+ 'timestamp', 'comments', 'message', 'likes', 'liked'])
1099 for column_name in empty_columns:
1100 self.assertEqual(row[COLUMN_INDICES[column_name]], '')
1101
1102
1103=== modified file 'gwibber/gwibber/tests/test_shortener.py'
1104--- gwibber/gwibber/tests/test_shortener.py 2012-08-31 18:55:10 +0000
1105+++ gwibber/gwibber/tests/test_shortener.py 2012-10-12 01:37:25 +0000
1106@@ -37,9 +37,10 @@
1107 from gwibber.shorteners import ur1ca
1108 from gwibber.shorteners import zima
1109 from gwibber.shorteners import lookup
1110-from gwibber.testing.mocks import FakeData, FakeOpen
1111-
1112-
1113+from gwibber.testing.mocks import FakeData, FakeOpen, FakeSoupMessage
1114+
1115+
1116+@mock.patch('gwibber.utils.download._soup', mock.Mock())
1117 class TestShorteners(unittest.TestCase):
1118 """Test the various shorteners, albeit via mocks."""
1119
1120@@ -132,8 +133,8 @@
1121 self.assertRaises(AttributeError,
1122 getattr, tinyurlcom.PROTOCOL_INFO, 'bogus')
1123
1124- @mock.patch('gwibber.utils.download.urlopen',
1125- FakeData('gwibber.tests.data', 'ur1ca.html'))
1126+ @mock.patch('gwibber.utils.download.Soup.Message',
1127+ FakeSoupMessage('gwibber.tests.data', 'ur1ca.html'))
1128 def test_ur1ca(self):
1129 # Test the shortener.
1130 self.assertEqual(
1131
1132=== modified file 'gwibber/gwibber/tests/test_twitter.py'
1133--- gwibber/gwibber/tests/test_twitter.py 2012-10-10 16:58:29 +0000
1134+++ gwibber/gwibber/tests/test_twitter.py 2012-10-12 01:37:25 +0000
1135@@ -27,7 +27,7 @@
1136
1137 from gwibber.protocols.twitter import RateLimiter, Twitter
1138 from gwibber.testing.helpers import FakeAccount
1139-from gwibber.testing.mocks import FakeData, LogMock
1140+from gwibber.testing.mocks import FakeSoupMessage, LogMock
1141 from gwibber.utils.model import COLUMN_TYPES
1142
1143
1144@@ -44,6 +44,7 @@
1145 TestModel.set_schema_full(COLUMN_TYPES)
1146
1147
1148+@mock.patch('gwibber.utils.download._soup', mock.Mock())
1149 class TestTwitter(unittest.TestCase):
1150 """Test the Twitter API."""
1151
1152@@ -107,8 +108,8 @@
1153 dict(Authorization=result))
1154
1155 @mock.patch('gwibber.utils.base.Model', TestModel)
1156- @mock.patch('gwibber.utils.download.urlopen',
1157- FakeData('gwibber.tests.data', 'twitter-home.dat'))
1158+ @mock.patch('gwibber.utils.download.Soup.Message',
1159+ FakeSoupMessage('gwibber.tests.data', 'twitter-home.dat'))
1160 @mock.patch('gwibber.protocols.twitter.Twitter._login',
1161 return_value=True)
1162 @mock.patch('gwibber.utils.base._seen_messages', {})
1163@@ -131,7 +132,7 @@
1164 'https://si0.twimg.com/profile_images/730275945/oauth-dancer_normal.jpg',
1165 'https://twitter.com/oauth_dancer/status/240558470661799936', '',
1166 '', '', '', 0.0, False, '', '', '', '', '', '', '', '', '', '',
1167- '', '', '', '', '', '', '', '', '', '', '',
1168+ '', '', '', '', '', '', '', [], '', '', '',
1169 ],
1170 [[['twitter', 'faker/than fake', '240556426106372096']],
1171 'messages', 'Raffi Krikorian', 'raffi', False,
1172@@ -140,7 +141,7 @@
1173 'https://si0.twimg.com/profile_images/1270234259/raffi-headshot-casual_normal.png',
1174 'https://twitter.com/raffi/status/240556426106372096', '',
1175 '', '', '', 0.0, False, '', '', '', '', '', '', '', '', '', '',
1176- '', '', '', '', '', '', '', '', '', '', '',
1177+ '', '', '', '', '', '', '', [], '', '', '',
1178 ],
1179 [[['twitter', 'faker/than fake', '240539141056638977']],
1180 'messages', 'Taylor Singletary', 'episod', False,
1181@@ -148,7 +149,7 @@
1182 'https://si0.twimg.com/profile_images/2546730059/f6a8zq58mg1hn0ha8vie_normal.jpeg',
1183 'https://twitter.com/episod/status/240539141056638977', '',
1184 '', '', '', 0.0, False, '', '', '', '', '', '', '', '', '', '',
1185- '', '', '', '', '', '', '', '', '', '', '',
1186+ '', '', '', '', '', '', '', [], '', '', '',
1187 ],
1188 ]
1189 for i, expected_row in enumerate(expected):
1190@@ -156,8 +157,8 @@
1191 self.assertEqual(got, want)
1192
1193 @mock.patch('gwibber.utils.base.Model', TestModel)
1194- @mock.patch('gwibber.utils.download.urlopen',
1195- FakeData('gwibber.tests.data', 'twitter-send.dat'))
1196+ @mock.patch('gwibber.utils.download.Soup.Message',
1197+ FakeSoupMessage('gwibber.tests.data', 'twitter-send.dat'))
1198 @mock.patch('gwibber.protocols.twitter.Twitter._login',
1199 return_value=True)
1200 @mock.patch('gwibber.utils.base._seen_messages', {})
1201@@ -181,7 +182,7 @@
1202 'https://si0.twimg.com/profile_images/730275945/oauth-dancer_normal.jpg',
1203 'https://twitter.com/oauth_dancer/status/240558470661799936', '',
1204 '', '', '', 0.0, False, '', '', '', '', '', '', '', '', '', '',
1205- '', '', '', '', '', '', '', '', '', '', '',
1206+ '', '', '', '', '', '', '', [], '', '', '',
1207 ]
1208 for got, want in zip(TestModel.get_row(0), expected_row):
1209 self.assertEqual(got, want)
1210@@ -378,9 +379,9 @@
1211 def test_rate_limiter_first_time(self, sleep):
1212 # The first time we see a URL, there is no rate limiting.
1213 limiter = RateLimiter()
1214- class Request:
1215- full_url = 'http://example.com'
1216- limiter.wait(Request())
1217+ message = FakeSoupMessage('gwibber.tests.data', 'twitter-home.dat')
1218+ message.new('GET', 'http://example.com/')
1219+ limiter.wait(message)
1220 sleep.assert_called_with(0)
1221
1222 @mock.patch('gwibber.protocols.twitter.time.sleep')
1223@@ -388,15 +389,14 @@
1224 def test_rate_limiter_second_time(self, time, sleep):
1225 # The second time we see the URL, we get rate limited.
1226 limiter = RateLimiter()
1227- class Request:
1228- full_url = 'http://example.com'
1229- response = FakeData('gwibber.tests.data', 'twitter-home.dat',
1230- headers={
1231- 'X-Rate-Limit-Reset': 1349382153 + 300,
1232- 'X-Rate-Limit-Remaining': 1,
1233- })
1234- limiter.update(response('http://example.com'))
1235- limiter.wait(Request())
1236+ message = FakeSoupMessage(
1237+ 'gwibber.tests.data', 'twitter-home.dat',
1238+ headers={
1239+ 'X-Rate-Limit-Reset': 1349382153 + 300,
1240+ 'X-Rate-Limit-Remaining': 1,
1241+ })
1242+ limiter.update(message.new('GET', 'http://example.com'))
1243+ limiter.wait(message)
1244 sleep.assert_called_with(300)
1245
1246 @mock.patch('gwibber.protocols.twitter.time.sleep')
1247@@ -404,15 +404,14 @@
1248 def test_rate_limiter_second_time_with_query(self, time, sleep):
1249 # A query parameter on the second request is ignored.
1250 limiter = RateLimiter()
1251- class Request:
1252- full_url = 'http://example.com/foo?baz=7'
1253- response = FakeData('gwibber.tests.data', 'twitter-home.dat',
1254- headers={
1255- 'X-Rate-Limit-Reset': 1349382153 + 300,
1256- 'X-Rate-Limit-Remaining': 1,
1257- })
1258- limiter.update(response('http://example.com/foo'))
1259- limiter.wait(Request())
1260+ message = FakeSoupMessage(
1261+ 'gwibber.tests.data', 'twitter-home.dat',
1262+ headers={
1263+ 'X-Rate-Limit-Reset': 1349382153 + 300,
1264+ 'X-Rate-Limit-Remaining': 1,
1265+ })
1266+ limiter.update(message.new('GET', 'http://example.com/foo?baz=7'))
1267+ limiter.wait(message)
1268 sleep.assert_called_with(300)
1269
1270 @mock.patch('gwibber.protocols.twitter.time.sleep')
1271@@ -420,15 +419,14 @@
1272 def test_rate_limiter_second_time_with_query_on_request(self, time, sleep):
1273 # A query parameter on the original request is ignored.
1274 limiter = RateLimiter()
1275- response = FakeData('gwibber.tests.data', 'twitter-home.dat',
1276- headers={
1277- 'X-Rate-Limit-Reset': 1349382153 + 300,
1278- 'X-Rate-Limit-Remaining': 1,
1279- })
1280- limiter.update(response('http://example.com/foo?baz=7'))
1281- class Request:
1282- full_url = 'http://example.com/foo'
1283- limiter.wait(Request())
1284+ message = FakeSoupMessage(
1285+ 'gwibber.tests.data', 'twitter-home.dat',
1286+ headers={
1287+ 'X-Rate-Limit-Reset': 1349382153 + 300,
1288+ 'X-Rate-Limit-Remaining': 1,
1289+ })
1290+ limiter.update(message.new('GET', 'http://example.com/foo?baz=7'))
1291+ limiter.wait(message)
1292 sleep.assert_called_with(300)
1293
1294 @mock.patch('gwibber.protocols.twitter.time.sleep')
1295@@ -437,15 +435,14 @@
1296 # With one remaining call this window, we get rate limited to the
1297 # full amount of the remaining window.
1298 limiter = RateLimiter()
1299- response = FakeData('gwibber.tests.data', 'twitter-home.dat',
1300- headers={
1301- 'X-Rate-Limit-Reset': 1349382153 + 300,
1302- 'X-Rate-Limit-Remaining': 1,
1303- })
1304- class Request:
1305- full_url = 'http://example.com/alpha'
1306- limiter.update(response(Request.full_url))
1307- limiter.wait(Request())
1308+ message = FakeSoupMessage(
1309+ 'gwibber.tests.data', 'twitter-home.dat',
1310+ headers={
1311+ 'X-Rate-Limit-Reset': 1349382153 + 300,
1312+ 'X-Rate-Limit-Remaining': 1,
1313+ })
1314+ limiter.update(message.new('GET', 'http://example.com/alpha'))
1315+ limiter.wait(message)
1316 sleep.assert_called_with(300)
1317
1318 @mock.patch('gwibber.protocols.twitter.time.sleep')
1319@@ -454,15 +451,14 @@
1320 # With no remaining calls left this window, we wait until the end of
1321 # the window.
1322 limiter = RateLimiter()
1323- response = FakeData('gwibber.tests.data', 'twitter-home.dat',
1324- headers={
1325- 'X-Rate-Limit-Reset': 1349382153 + 300,
1326- 'X-Rate-Limit-Remaining': 0,
1327- })
1328- class Request:
1329- full_url = 'http://example.com/alpha'
1330- limiter.update(response(Request.full_url))
1331- limiter.wait(Request())
1332+ message = FakeSoupMessage(
1333+ 'gwibber.tests.data', 'twitter-home.dat',
1334+ headers={
1335+ 'X-Rate-Limit-Reset': 1349382153 + 300,
1336+ 'X-Rate-Limit-Remaining': 0,
1337+ })
1338+ limiter.update(message.new('GET', 'http://example.com/alpha'))
1339+ limiter.wait(message)
1340 sleep.assert_called_with(300)
1341
1342 @mock.patch('gwibber.protocols.twitter.time.sleep')
1343@@ -471,15 +467,14 @@
1344 # With a few calls remaining this window, we time slice the remaining
1345 # time evenly between those remaining calls.
1346 limiter = RateLimiter()
1347- response = FakeData('gwibber.tests.data', 'twitter-home.dat',
1348- headers={
1349- 'X-Rate-Limit-Reset': 1349382153 + 300,
1350- 'X-Rate-Limit-Remaining': 3,
1351- })
1352- class Request:
1353- full_url = 'http://example.com/beta'
1354- limiter.update(response(Request.full_url))
1355- limiter.wait(Request())
1356+ message = FakeSoupMessage(
1357+ 'gwibber.tests.data', 'twitter-home.dat',
1358+ headers={
1359+ 'X-Rate-Limit-Reset': 1349382153 + 300,
1360+ 'X-Rate-Limit-Remaining': 3,
1361+ })
1362+ limiter.update(message.new('GET', 'http://example.com/beta'))
1363+ limiter.wait(message)
1364 sleep.assert_called_with(100.0)
1365
1366 @mock.patch('gwibber.protocols.twitter.time.sleep')
1367@@ -488,26 +483,27 @@
1368 # With more than 5 calls remaining in this window, we don't rate
1369 # limit, even if we've already seen this url.
1370 limiter = RateLimiter()
1371- response = FakeData('gwibber.tests.data', 'twitter-home.dat',
1372- headers={
1373- 'X-Rate-Limit-Reset': 1349382153 + 300,
1374- 'X-Rate-Limit-Remaining': 10,
1375- })
1376- class Request:
1377- full_url = 'http://example.com/omega'
1378- limiter.update(response(Request.full_url))
1379- limiter.wait(Request())
1380+ message = FakeSoupMessage(
1381+ 'gwibber.tests.data', 'twitter-home.dat',
1382+ headers={
1383+ 'X-Rate-Limit-Reset': 1349382153 + 300,
1384+ 'X-Rate-Limit-Remaining': 10,
1385+ })
1386+ limiter.update(message.new('GET', 'http://example.com/omega'))
1387+ limiter.wait(message)
1388 sleep.assert_called_with(0)
1389
1390 @mock.patch('gwibber.utils.base.Model', TestModel)
1391 @mock.patch('gwibber.protocols.twitter.Twitter._login',
1392 return_value=True)
1393- @mock.patch('gwibber.utils.download.urlopen',
1394- FakeData('gwibber.tests.data', 'twitter-home.dat',
1395- headers={
1396- 'X-Rate-Limit-Reset': 1349382153 + 300,
1397- 'X-Rate-Limit-Remaining': 3,
1398- }))
1399+ @mock.patch(
1400+ 'gwibber.utils.download.Soup.Message',
1401+ FakeSoupMessage(
1402+ 'gwibber.tests.data', 'twitter-home.dat',
1403+ headers={
1404+ 'X-Rate-Limit-Reset': 1349382153 + 300,
1405+ 'X-Rate-Limit-Remaining': 3,
1406+ }))
1407 @mock.patch('gwibber.protocols.twitter.time.sleep')
1408 @mock.patch('gwibber.protocols.twitter.time.time', return_value=1349382153)
1409 def test_protocol_rate_limiting(self, time, sleep, login):
1410
1411=== modified file 'gwibber/gwibber/utils/base.py'
1412--- gwibber/gwibber/utils/base.py 2012-10-10 14:33:01 +0000
1413+++ gwibber/gwibber/utils/base.py 2012-10-12 01:37:25 +0000
1414@@ -297,7 +297,7 @@
1415
1416 result = Authentication(self._account, log).login()
1417 if result is None:
1418- log.error('No {} authentication results received'.format(protocol))
1419+ log.error('No {} authentication results received.'.format(protocol))
1420 return
1421
1422 token = result.get('AccessToken')
1423
1424=== modified file 'gwibber/gwibber/utils/download.py'
1425--- gwibber/gwibber/utils/download.py 2012-10-05 01:19:29 +0000
1426+++ gwibber/gwibber/utils/download.py 2012-10-12 01:37:25 +0000
1427@@ -26,13 +26,26 @@
1428
1429 from base64 import encodebytes
1430 from contextlib import contextmanager
1431+from gi.repository import Soup, SoupGNOME
1432+from urllib.error import HTTPError
1433 from urllib.parse import urlencode
1434-from urllib.request import Request, urlopen
1435-
1436
1437 log = logging.getLogger('gwibber.service')
1438
1439
1440+# Global libsoup session instance.
1441+_soup = Soup.SessionSync()
1442+# Enable this for full requests and responses dumped to STDOUT.
1443+#_soup.add_feature(Soup.Logger.new(Soup.LoggerLogLevel.BODY, -1))
1444+_soup.add_feature(SoupGNOME.ProxyResolverGNOME())
1445+
1446+
1447+def _get_charset(message):
1448+ """Extract charset from Content-Type header in a Soup Message."""
1449+ type_header = message.response_headers.get_content_type()[1] or {}
1450+ return type_header.get('charset')
1451+
1452+
1453 class RateLimiter:
1454 """Base class for the rate limiting API.
1455
1456@@ -40,22 +53,23 @@
1457 override the `wait()` and `update()` methods for protocol specific
1458 rate-limiting functionality.
1459 """
1460- def wait(self, request):
1461+ def wait(self, message):
1462 """Wait an appropriate amount of time before returning.
1463
1464 Downloading is blocked until this method returns. This does not block
1465 the entire application since downloading always happens in a
1466 sub-thread. If no wait is necessary, return immediately.
1467
1468- :param request: The soon-to-be executed url request.
1469- :type request: urllib.request.Request
1470+ :param message: The constructed but unsent libSoup Message.
1471+ :type message: Soup.Message
1472 """
1473 pass
1474
1475- def update(self, response):
1476+ def update(self, message):
1477 """Update any rate limiting values based on the service's response.
1478
1479- :param response: A urllib response object.
1480+ :param message: The same libSoup Message complete with response headers.
1481+ :type message: Soup.Message
1482 """
1483 pass
1484
1485@@ -63,84 +77,101 @@
1486 class Downloader:
1487 """Convenient downloading wrapper."""
1488
1489- def __init__(self, url, params=None, post=False,
1490+ def __init__(self, url, params=None, method='GET',
1491 username=None, password=None,
1492 headers=None, rate_limiter=None):
1493 self.url = url
1494- self.params = params
1495- self.post = post
1496+ self.method = method
1497 self.username = username
1498 self.password = password
1499+ self.params = ({} if params is None else params)
1500 self.headers = ({} if headers is None else headers)
1501 self._rate_limiter = (RateLimiter() if rate_limiter is None
1502 else rate_limiter)
1503 # XXX Proxy handling.
1504
1505 def _build_request(self):
1506- """Return a request object, with all the right headers.
1507+ """Return a libsoup message, with all the right headers.
1508
1509- :return: The request.
1510- :rtype: urllib.request.Request
1511+ :return: A constructed but unsent libsoup message.
1512+ :rtype: Soup.Message
1513 """
1514 data = None
1515 url = self.url
1516- headers = self.headers.copy()
1517- if self.params is not None:
1518- # urlencode() does not have an option to use quote() instead of
1519- # quote_plus(), but Twitter requires percent-encoded spaces, and
1520- # this is harmless to any other protocol.
1521- params = urlencode(self.params).replace('+', '%20')
1522- if self.post:
1523- data = params.encode('utf-8')
1524- headers['Content-Type'] = (
1525- 'application/x-www-form-urlencoded; charset=utf-8')
1526- else:
1527+
1528+ # urlencode() does not have an option to use quote() instead of
1529+ # quote_plus(), but Twitter requires percent-encoded spaces, and
1530+ # this is harmless to any other protocol.
1531+ params = urlencode(self.params).replace('+', '%20')
1532+ if params:
1533+ if self.method == 'GET':
1534 # Do a GET with an encoded query string.
1535 url = '{}?{}'.format(self.url, params)
1536- request = Request(url, data, headers)
1537+ else:
1538+ data = params
1539+
1540+ message = Soup.Message.new(self.method, url)
1541+ for (header, value) in self.headers.items():
1542+ message.request_headers.append(header, value)
1543+ if data is not None:
1544+ message.set_request(
1545+ 'application/x-www-form-urlencoded; charset=utf-8',
1546+ Soup.MemoryUse.COPY, data, len(data))
1547+
1548 if self.username is not None and self.password is not None:
1549 auth = '{}:{}'.format(self.username, self.password).encode('utf-8')
1550 # encodebytes() includes a bogus trailing newline, which we must
1551 # strip off.
1552 value = encodebytes(auth)[:-1].decode('utf-8')
1553 basic = 'Basic {}'.format(value)
1554- request.add_header('Authorization', basic)
1555+ message.request_headers.append('Authorization', basic)
1556+
1557 # Possibly do some rate limiting.
1558- self._rate_limiter.wait(request)
1559- return request
1560+ self._rate_limiter.wait(message)
1561+ return message
1562
1563 @contextmanager
1564 def _download(self):
1565 """Perform the download, as a context manager."""
1566- request = self._build_request()
1567- with urlopen(request) as result:
1568- yield result
1569- self._rate_limiter.update(result)
1570+ message = self._build_request()
1571+ _soup.send_message(message)
1572+ if message.status_code != 200:
1573+ raise HTTPError(self.url,
1574+ message.status_code,
1575+ message.reason_phrase,
1576+ self.headers,
1577+ None)
1578+ # log.error(
1579+ # '{}: {} {}: {}'.format(
1580+ # message.get_uri().host,
1581+ # message.status_code,
1582+ # message.reason_phrase,
1583+ # message.response_body.flatten().get_data()))
1584+ yield message
1585+ self._rate_limiter.update(message)
1586
1587 def get_bytes(self):
1588 """Return the results as a bytes object."""
1589- with self._download() as result:
1590- return result.read()
1591+ with self._download() as message:
1592+ return message.response_body.flatten().get_data()
1593
1594 def get_string(self):
1595 """Return the results as a string, decoded as per the response."""
1596- with self._download() as result:
1597- payload = result.read()
1598- encoding = result.info().get_content_charset()
1599- return payload.decode(encoding)
1600+ with self._download() as message:
1601+ payload = message.response_body.flatten().get_data()
1602+ return payload.decode(_get_charset(message))
1603
1604 def get_json(self):
1605 """Interpret and return the results as JSON data."""
1606- with self._download() as result:
1607- payload = result.read()
1608- info = result.info()
1609+ with self._download() as message:
1610+ payload = message.response_body.flatten().get_data()
1611+ charset = _get_charset(message)
1612 # RFC 4627 $3. JSON text SHALL be encoded in Unicode. The default
1613 # encoding is UTF-8. Since the first two characters of a JSON text
1614 # will always be ASCII characters [RFC0020], it is possible to
1615 # determine whether an octet stream is UTF-8, UTF-16 (BE or LE), or
1616 # UTF-32 (BE or LE) by looking at the pattern of nulls in the first
1617 # four octets.
1618- charset = info.get_content_charset()
1619 if charset is None:
1620 octet_0, octet_1, octet_2, octet_3 = payload[:4]
1621 if 0 not in (octet_0, octet_1, octet_2, octet_3):
1622
1623=== modified file 'gwibber/gwibber/utils/model.py'
1624--- gwibber/gwibber/utils/model.py 2012-09-20 16:26:53 +0000
1625+++ gwibber/gwibber/utils/model.py 2012-10-12 01:37:25 +0000
1626@@ -86,7 +86,7 @@
1627 ('video_src', 's'),
1628 ('video_url', 's'),
1629 ('video_name', 's'),
1630- ('comments', 's'),
1631+ ('comments', 'as'),
1632 ('recipient', 's'),
1633 ('recipient_nick', 's'),
1634 ('recipient_icon', 's'),
1635@@ -99,7 +99,12 @@
1636 # pulling column values out of a row of data.
1637 COLUMN_INDICES = {name: i for i, name in enumerate(COLUMN_NAMES)}
1638 # This defines default values for the different data types
1639-DEFAULTS = dict(s='', b=False, d=0)
1640+DEFAULTS = {
1641+ 'as': [],
1642+ 'b': False,
1643+ 's': '',
1644+ 'd': 0,
1645+ }
1646
1647
1648 Model = Dee.SharedModel.new('com.Gwibber.Streams')
1649
1650=== removed file 'gwibber/microblog/__init__.py'
1651=== removed directory 'gwibber/microblog/plugins'
1652=== removed file 'gwibber/microblog/plugins/__init__.py'
1653=== removed directory 'gwibber/microblog/plugins/facebook'
1654=== removed file 'gwibber/microblog/plugins/facebook/__init__.py'
1655--- gwibber/microblog/plugins/facebook/__init__.py 2012-09-25 20:40:09 +0000
1656+++ gwibber/microblog/plugins/facebook/__init__.py 1970-01-01 00:00:00 +0000
1657@@ -1,480 +0,0 @@
1658-#!/usr/bin/env python
1659-
1660-from gwibber.microblog import network, util
1661-from gwibber.microblog.util import resources
1662-import hashlib, time
1663-from gettext import lgettext as _
1664-from gwibber.microblog.util.auth import Authentication
1665-from gwibber.microblog.util.const import *
1666-# Try to import * from custom, install custom.py to include packaging
1667-# customizations like distro API keys, etc
1668-try:
1669- from gwibber.microblog.util.custom import *
1670-except:
1671- pass
1672-from gwibber.microblog.util.time import parsetime
1673-
1674-from datetime import datetime, timedelta
1675-
1676-import logging
1677-logger = logging.getLogger("Facebook")
1678-logger.debug("Initializing.")
1679-
1680-import json, urllib
1681-
1682-PROTOCOL_INFO = {
1683- "name": "Facebook",
1684- "version": "1.1",
1685-
1686- "config": [
1687- "color",
1688- "receive_enabled",
1689- "send_enabled",
1690- "username",
1691- "uid",
1692- "private:access_token"
1693- ],
1694-
1695- "authtype": "facebook",
1696- "color": "#64006C",
1697-
1698- "features": [
1699- "send",
1700- "reply",
1701- "receive",
1702- "thread",
1703- "delete",
1704- "send_thread",
1705- "like",
1706- "unlike",
1707- "sincetime"
1708- ],
1709-
1710- "default_streams": [
1711- "receive",
1712- "images",
1713- "links",
1714- "videos",
1715- ]
1716-}
1717-
1718-URL_PREFIX = "https://graph.facebook.com/"
1719-POST_URL = "http://www.facebook.com/profile.php?id=%s&v=feed&story_fbid=%s&ref=mf"
1720-
1721-
1722-def convert_time(data):
1723- """Extract and convert the time to a timestamp (seconds since Epoch).
1724-
1725- First, look for the 'updated_time' key in the data, falling back to
1726- 'created_time' if the former is missing.
1727-
1728- """
1729- # Convert from an ISO 8601 format time string to a Epoch timestamp.
1730- # Assume standard ISO 8601 'T' separator, no microseconds, and naive
1731- # time zone as per:
1732- # https://developers.facebook.com/docs/reference/api/event/
1733- time_string = data.get('updated_time', data['created_time'])
1734- return parsetime(time_string)
1735-
1736-
1737-class Client:
1738- """Query Facebook and convert data.
1739-
1740- The Client class is responsible for querying Facebook and turning the data obtained
1741- into data that Gwibber can understand. Facebook uses a version of oauth for security.
1742- Tokens have already been obtained when the account was set up in Gwibber and are used to
1743- authenticate when getting data.
1744-
1745- """
1746- def __init__(self, acct):
1747- self.account = acct
1748- self.user_id = acct.get("uid", None)
1749- self._loop = None
1750-
1751- def _check_error(self, data):
1752- """Checks if data is from the correct form.
1753-
1754- Checks to ensure the data obtained by Facebook is on the correct form.
1755- If not, an error is logged in gwibber.log
1756-
1757- Arguments:
1758- data -- A data structure obtained from Foursquare
1759-
1760- Returns:
1761- True if data is contains an error or is not a dictionary, otherwise, False.
1762-
1763- """
1764- if isinstance(data, dict):
1765- if data.has_key("error"):
1766- logger.info("Facebook error %s - %s", data["error"]["type"], data["error"]["message"])
1767- return True
1768- else:
1769- return False
1770- return True
1771-
1772- def _retrieve_user_id(self):
1773- data = json.loads(urllib.urlopen("https://graph.facebook.com/me?access_token=" + self.account["access_token"]).read())
1774- if isinstance(data, dict):
1775- if data.has_key("id") and data.has_key("name"):
1776- self.account["username"] = data["name"]
1777- self.account["uid"] = data["id"]
1778- else:
1779- # Make a desparate attempt to guess the id from the url
1780- uid = url.split('-')[1].split('%7C')[0]
1781- if isinstance(uid, int) and len(uid) > 2:
1782- acct = json.loads(urllib.urlopen("https://graph.facebook.com/" + str(uid)).read())
1783- if isinstance(acct, dict):
1784- if acct.has_key("id") and acct.has_key("name"):
1785- self.account["uid"] = acct["id"]
1786- self.account["username"] = acct["name"]
1787- else:
1788- logger.error("Couldn't get facebook user ID")
1789- if "uid" in self.account:
1790- self.user_id = self.account["uid"]
1791-
1792- def _login(self):
1793- old_token = self.account.get("access_token", None)
1794- with self.account.login_lock:
1795- # Perform the login only if it wasn't already performed by another thread
1796- # while we were waiting to the lock
1797- if self.account.get("access_token", None) == old_token:
1798- self._locked_login(old_token)
1799-
1800- return "access_token" in self.account and \
1801- self.account["access_token"] != old_token
1802-
1803- def _locked_login(self, old_token):
1804- logger.debug("Re-authenticating" if old_token else "Logging in")
1805-
1806- auth = Authentication(self.account, logger)
1807- reply = auth.login()
1808- if reply and reply.has_key("AccessToken"):
1809- self.account["access_token"] = reply["AccessToken"]
1810- self._retrieve_user_id()
1811- logger.debug("User id is: %s" % self.account["uid"])
1812- else:
1813- logger.error("Didn't find token in session: %s", (reply,))
1814-
1815- def _get(self, operation, post=False, single=False, **args):
1816- """Establishes a connection with Facebook and gets the data requested.
1817-
1818- Arguments:
1819- operation -- The end of the URL to look up on Facebook
1820- post -- True is this is a send, False if a receive (False by default)
1821- single -- True if a single checkin is requested, False if multiple (False by default)
1822- **args -- Arguments to be added to the URL when accessed.
1823-
1824- Returns:
1825- A list of Facebook objects.
1826-
1827- """
1828- if "access_token" not in self.account and not self._login():
1829- logstr = """%s: %s - %s""" % (PROTOCOL_INFO["name"], _("Authentication failed"), "Auth needs updating")
1830- logger.error("%s", logstr)
1831- return [{"error": {"type": "auth", "account": self.account, "message": _("Authentication failed, please re-authorize")}}]
1832-
1833- args.update({
1834- "access_token": self.account["access_token"]
1835- })
1836-
1837- sig = "".join("%s=%s" % (k, v) for k, v in sorted(args.items()))
1838- args["sig"] = hashlib.md5(sig + self.account["access_token"]).hexdigest()
1839- data = network.Download((URL_PREFIX + operation), args, post).get_json()
1840-
1841- resources.dump(self.account["service"], self.account["id"], data)
1842-
1843- if isinstance(data, dict) and data.get("error_msg", 0):
1844- if "permission" in data["error_msg"]:
1845- # Try again, if we get a new token
1846- if self._login():
1847- return self._get(operation, post, single, args)
1848- else:
1849- logstr = """%s: %s - %s""" % (PROTOCOL_INFO["name"], _("Authentication failed"), data["error_msg"])
1850- logger.error("%s", logstr)
1851- return [{"error": {"type": "auth", "account": self.account, "message": data["error_msg"]}}]
1852- elif data["error_code"] == 102:
1853- logstr = """%s: %s - %s""" % (PROTOCOL_INFO["name"], _("Session invalid"), data["error_msg"])
1854- logger.error("%s", logstr)
1855- return [{"error": {"type": "auth", "account": self.account, "message": data["error_msg"]}}]
1856- else:
1857- logstr = """%s: %s - %s""" % (PROTOCOL_INFO["name"], _("Unknown failure"), data["error_msg"])
1858- logger.error("%s", logstr)
1859- return [{"error": {"type": "unknown", "account": self.account, "message": data["error_msg"]}}]
1860-
1861- return data
1862-
1863- def _sender(self, data):
1864- """Parses the sender of the Facebook object.
1865-
1866- Arguments:
1867- data -- A Facebook object
1868-
1869- Returns:
1870- A sender object in a format compatible with Gwibber's database.
1871-
1872- """
1873- sender = {
1874- "name": data["name"],
1875- "id": str(data.get("id", '')),
1876- "is_me": str(data.get("id", '')) == self.user_id,
1877- "image": URL_PREFIX + data["id"] + "/picture?type=large" ,
1878- "url": "https://www.facebook.com/profile.php?id=" + str(data.get("id", ''))
1879- }
1880- return sender
1881-
1882- def _message(self, data):
1883- """Parses a Facebook object.
1884-
1885- Arguments:
1886- data -- A Facebook object
1887-
1888- Returns:
1889- A Facebook object in a format compatible with Gwibber's database.
1890-
1891- """
1892- if type(data) != dict:
1893- logger.error("Cannot parse message data: %s", str(data))
1894- return {}
1895-
1896- m = {}
1897- m["mid"] = str(data["id"])
1898- m["service"] = "facebook"
1899- m["account"] = self.account["id"]
1900-
1901- m["time"] = convert_time(data)
1902- m["url"] = "https://facebook.com/" + data["id"].split("_")[0] + "/posts/" + data["id"].split("_")[1]
1903-
1904-
1905- if data.get("attribution", 0):
1906- m["source"] = util.strip_urls(data["attribution"]).replace("via ", "")
1907-
1908- if data.has_key("message"):
1909- m["to_me"] = ("@%s" % self.account["username"]) in data["message"]
1910- if data.get("message", "").strip():
1911- m["text"] = data["message"]
1912- m["html"] = util.linkify(data["message"])
1913- m["content"] = m["html"]
1914- else:
1915- m["text"] = ""
1916- m["html"] = ""
1917- m["content"] = ""
1918-
1919- m["sender"] = self._sender(data["from"])
1920-
1921- m["type"] = data["type"]
1922-
1923- if data["type"] == "checkin":
1924- m["location"] = {}
1925- if isinstance(data["place"], dict):
1926- m["location"]["id"] = data["place"]["id"]
1927- m["location"]["name"] = data["place"].get("name", None)
1928- if m["location"]["name"]:
1929- m["html"] += " " + _("at").decode('utf8') + " <p><b>%s</b></p>" % m["location"]["name"]
1930- m["content"] = m["html"]
1931- m["text"] += " " + _("at").decode('utf8') + " " + m["location"]["name"]
1932- if data["place"].has_key("location"):
1933- m["location"]["latitude"] = data["place"]["location"].get("latitude", None)
1934- m["location"]["longitude"] = data["place"]["location"].get("longitude", None)
1935- m["location"]["city"] = data["place"]["location"].get("city", None)
1936- m["location"]["state"] = data["place"]["location"].get("state", None)
1937- m["location"]["country"] = data["place"]["location"].get("country", None)
1938- elif data["type"] == "photo":
1939- m["photo"] = {}
1940- m["photo"]["picture"] = data.get("picture", None)
1941- m["photo"]["url"] = data.get("link", None)
1942- m["photo"]["name"] = data.get("name", None)
1943- elif data["type"] == "video":
1944- m["video"] = {}
1945- m["video"]["picture"] = data.get("picture", None)
1946- m["video"]["source"] = data.get("source", None)
1947- m["video"]["url"] = data.get("link", None)
1948- m["video"]["name"] = data.get("name", None)
1949- m["video"]["icon"] = data.get("icon", None)
1950- m["video"]["properties"] = data.get("properties", {})
1951- elif data["type"] == "link":
1952- m["link"] = {}
1953- m["link"]["picture"] = data.get("picture", None)
1954- m["link"]["name"] = data.get("name", None)
1955- m["link"]["description"] = data.get("description", None)
1956- m["link"]["url"] = data.get("link", None)
1957- m["link"]["icon"] = data.get("icon", None)
1958- m["link"]["caption"] = data.get("caption", None)
1959- m["link"]["properties"] = data.get("properties", {})
1960- elif data["type"] == "question":
1961- m["question"] = {}
1962- m["question"]["id"] = data["id"]
1963- m["question"]["text"] = data.get("question", None)
1964- if not m["question"]["text"]:
1965- m["question"]["text"] = data.get("story", None)
1966- if data.has_key("options") and isinstance(data["options"], dict):
1967- if isinstance(data["options"]["data"], dict):
1968- m["question"]["answers"] = []
1969- for a in data["options"]["data"]:
1970- answer = {}
1971- answer["id"] = a.get("id", None)
1972- answer["name"] = a.get("name", None)
1973- answer["votes"] = a.get("votes", None)
1974- m["question"]["answers"].append(answer)
1975- if m["question"]["text"] and len(m["text"]) < 1:
1976- m["text"] = m["question"]["text"]
1977- m["html"] = m["text"]
1978- m["content"] = m["html"]
1979- elif data["type"] == "status":
1980- pass
1981- else:
1982- logger.error ("facebook: unexpected type %s", data["type"])
1983-
1984- if data.has_key("privacy"):
1985- m["privacy"] = {}
1986- m["privacy"]["description"] = data["privacy"]["description"]
1987- m["privacy"]["value"] = data["privacy"]["value"]
1988-
1989- # Handle target for wall posts with a specific recipient
1990- if data.has_key("to"):
1991- m["sender"]["name"] += " \u25b8 %s"%(data["to"]["data"][0]["name"])
1992-
1993- if data.has_key("likes"):
1994- m["likes"] = {}
1995- m["likes"]["liked"] = False
1996- if isinstance(data["likes"], dict):
1997- m["likes"]["count"] = data["likes"]["count"]
1998- if data["likes"].has_key("data"):
1999- m["likes"]["data"] = data["likes"]["data"]
2000- for d in m["likes"]["data"]:
2001- if self.user_id == str(d["id"]):
2002- m["likes"]["liked"] = True
2003- else:
2004- m["likes"]["count"] = data["likes"]
2005-
2006- if data.get("comments", 0):
2007- m["comments"] = []
2008- if data["comments"].has_key("data"):
2009- for item in data["comments"]["data"]:
2010- m["comments"].append({
2011- "text": item["message"],
2012- "time": convert_time(data),
2013- "sender": self._sender(item["from"]),
2014- })
2015-
2016- if data.get("attachment", 0):
2017- if data["attachment"].get("name", 0):
2018- m["content"] += "<p><b>%s</b></p>" % data["attachment"]["name"]
2019-
2020- if data["attachment"].get("description", 0):
2021- m["content"] += "<p>%s</p>" % data["attachment"]["description"]
2022-
2023- return m
2024-
2025- def __call__(self, opname, **args):
2026- return getattr(self, opname)(**args)
2027-
2028- def receive(self, since=None):
2029- """Gets a list of Facebook objects and add them to the database.
2030-
2031- Arguments:
2032- since -- The time to get Facebook objects from
2033-
2034- Returns:
2035- A list of Gwibber compatible objects which have been parsed by the parse function.
2036-
2037- """
2038- if since is None:
2039- since = (datetime.now() - timedelta(hours=240.0)).isoformat()
2040- else:
2041- since = datetime.fromtimestamp(since).isoformat()
2042-
2043- data = self._get("me/home", since=since, limit=100)
2044-
2045- logger.debug("<STATS> facebook:receive account:%s since:%s size:%s",
2046- self.account["id"], str(since), len(str(data)))
2047-
2048- if not self._check_error(data):
2049- try:
2050- return [self._message(post) for post in data["data"]]
2051- except TypeError:
2052- logger.error("<facebook:receive> failed to parse message data")
2053- return data
2054- else: return
2055-
2056- def delete(self, message):
2057- """Deletes a specified message from Facebook.
2058-
2059- Arguments:
2060- message -- A Gwibber compatible message object (from Gwibber's database)
2061-
2062- Returns:
2063- Nothing
2064-
2065- """
2066- result = self._get("stream.remove", post_id=message["mid"])
2067- if not result:
2068- logger.error("<facebook:delete> failed")
2069- return
2070- return []
2071-
2072- def like(self, message):
2073- """Likes a specified message on Facebook.
2074-
2075- Arguments:
2076- message -- a Gwibber compatible message object (from Gwibber's database)
2077-
2078- Returns:
2079- The liked message
2080-
2081- """
2082- result = self._get(message["mid"] + "/likes", post=True)
2083- if not result:
2084- logger.error("<facebook:like> failed")
2085- return []
2086- data = self.receive ()
2087- return data
2088-
2089- def unlike(self, message):
2090- """Unlikes a specified message on Facebook.
2091-
2092- Arguments:
2093- message -- A Gwibber compatible message object (from Gwibber's database)
2094-
2095- Returns:
2096- The unliked message
2097-
2098- """
2099- result = self._get(message["mid"] + "/likes", post=True, method="delete")
2100- if not result:
2101- logger.error("<facebook:like> failed")
2102- return []
2103- data = self.receive ()
2104- return data
2105-
2106- def send(self, message):
2107- """Sends a message to Facebook.
2108-
2109- Arguments:
2110- message -- The text of the message
2111-
2112- Returns:
2113- Nothing
2114-
2115- """
2116- result = self._get("me/feed", message=message, status_includes_verb=True, post=True)
2117- if not result:
2118- logger.error("<facebook:send> failed")
2119- return
2120- return []
2121-
2122- def send_thread(self, message, target):
2123- """Sends a message to Facebook as a reply to another message.
2124-
2125- Arguments:
2126- message -- The text of the message
2127- target -- The person the message is directed at (as a Gwibber compatible object)
2128-
2129- Returns:
2130- Nothing
2131-
2132- """
2133- result = self._get(target["mid"] + "/comments", message=message, post=True)
2134- if not result:
2135- logger.error("<facebook:send_thread> failed")
2136- return
2137- return []
2138
2139=== removed directory 'gwibber/microblog/plugins/statusnet'
2140=== removed file 'gwibber/microblog/plugins/statusnet/__init__.py'
2141--- gwibber/microblog/plugins/statusnet/__init__.py 2012-06-12 19:52:43 +0000
2142+++ gwibber/microblog/plugins/statusnet/__init__.py 1970-01-01 00:00:00 +0000
2143@@ -1,589 +0,0 @@
2144-from gettext import lgettext as _
2145-from oauth import oauth
2146-
2147-from gwibber.microblog import network, util
2148-from gwibber.microblog.util import resources
2149-from gwibber.microblog.util.time import parsetime
2150-
2151-import logging
2152-logger = logging.getLogger("StatusNet")
2153-logger.debug("Initializing.")
2154-
2155-PROTOCOL_INFO = {
2156- "name": "StatusNet",
2157- "version": 1.1,
2158-
2159- "config": [
2160- "private:secret_token",
2161- "access_token",
2162- "username",
2163- "site_display_name",
2164- "url_prefix",
2165- "color",
2166- "receive_enabled",
2167- "send_enabled",
2168- ],
2169-
2170- "authtype": "oauth1a",
2171- "color": "#4E9A06",
2172-
2173- "features": [
2174- "send",
2175- "receive",
2176- "reply",
2177- "responses",
2178- "private",
2179- "public",
2180- "delete",
2181- "follow",
2182- "unfollow",
2183- "profile",
2184- "retweet",
2185- "like",
2186- "send_thread",
2187- "send_private",
2188- "user_messages",
2189- "sinceid",
2190- ],
2191-
2192- "default_streams": [
2193- "receive",
2194- "images",
2195- "responses",
2196- "private",
2197- "public",
2198- ],
2199-}
2200-
2201-class Client:
2202- """Query the StatusNet server and convert the data.
2203-
2204- The Client class is responsible for querying the StatusNet server and turning the data obtained
2205- into data that Gwibber can understand. The StatusNet server uses a version of OAuth for security.
2206-
2207- Tokens have already been obtained when the account was set up in Gwibber and are used to
2208- authenticate when getting data.
2209-
2210- """
2211- def __init__(self, acct):
2212- if acct.has_key("url_prefix"):
2213- pref = "" if acct["url_prefix"].startswith("http") else "https://"
2214- self.url_prefix = pref + acct["url_prefix"]
2215-
2216- if acct.has_key("secret_token") and acct.has_key("password"): acct.pop("password")
2217- if not acct.has_key("url_prefix") and acct.has_key("domain"): acct.pop("domain")
2218- self.account = acct
2219-
2220- def _common(self, data):
2221- """Parses messages into Gwibber compatible forms.
2222-
2223- This function is common to all update types
2224- and includes parsing of user mentions, hashtags,
2225- urls, images and videos.
2226-
2227- Arguments:
2228- data -- A data object obtained from the StatusNet server containing a complete update
2229-
2230- Returns:
2231- m -- A data object compatible with inserting into the Gwibber database for that update
2232-
2233- """
2234- m = {}
2235- try:
2236- m["mid"] = str(data["id"])
2237- m["service"] = "statusnet"
2238- m["account"] = self.account["id"]
2239- if data.has_key("created_at"):
2240- m["time"] = parsetime(data["created_at"])
2241- m["source"] = data.get("source", False)
2242- m["text"] = util.unescape(data["text"])
2243- m["to_me"] = ("@%s" % self.account["username"]) in data["text"]
2244-
2245- m["html"] = util.linkify(m["text"],
2246- ((util.PARSE_HASH, '#<a class="hash" href="%s#search?q=\\1">\\1</a>' % self.account["url_prefix"]),
2247- (util.PARSE_NICK, '@<a class="nick" href="%s/\\1">\\1</a>' % self.account["url_prefix"])), escape=False)
2248-
2249- m["content"] = util.linkify(m["text"],
2250- ((util.PARSE_HASH, '#<a href="gwibber:/tag?acct=%s&query=\\1">\\1</a>' % m["account"]),
2251- (util.PARSE_NICK, '@<a href="gwibber:/user?acct=%s&name=\\1">\\1</a>' % m["account"])), escape=True)
2252-
2253- m["favorited"] = data.get("favorited", False)
2254-
2255- images = []
2256- if data.get("attachments", 0):
2257- for a in data["attachments"]:
2258- mime = a.get("mimetype", "")
2259- if mime and mime.startswith("image") and a.get("url", 0):
2260- images.append({"src": a["url"], "url": a["url"]})
2261-
2262- images.extend(util.imgpreview(m["text"]))
2263-
2264- if images:
2265- m["images"] = images
2266- m["type"] = "photo"
2267- except:
2268- logger.error("%s failure - %s", PROTOCOL_INFO["name"], data)
2269-
2270- return m
2271-
2272- def _user(self, user):
2273- """Parses the user portion of an update.
2274-
2275- Arguments:
2276- user -- A user object from an update
2277-
2278- Returns:
2279- A user object in a format compatible with Gwibber's database
2280-
2281- """
2282- return {
2283- "name": user.get("name", None),
2284- "nick": user.get("screen_name", None),
2285- "id": user.get("id", None),
2286- "location": user.get("location", None),
2287- "followers": user.get("followers_count", None),
2288- "friends": user.get("friends_count", None),
2289- "description": user.get("description", None),
2290- "following": user.get("following", None),
2291- "protected": user.get("protected", None),
2292- "statuses": user.get("statuses_count", None),
2293- "image": user.get("profile_image_url", None),
2294- "website": user.get("url", None),
2295- "url": "/".join((self.url_prefix, user["screen_name"])) or None,
2296- "is_me": user.get("screen_name", None) == self.account["username"],
2297- }
2298-
2299- def _message(self, data):
2300- """Parses messages into Gwibber compatible forms.
2301-
2302- This is the initial function for tweet parsing and parses
2303- shared status, source (the program the update was sent from)
2304- and reply details (the in-reply-to portion). It sends the rest
2305- to _common() for further parsing.
2306-
2307- Arguments:
2308- data -- A data object obtained from the StatusNet server containing a complete update
2309-
2310- Returns:
2311- m -- A data object compatible with inserting into the Gwibber database for that tweet
2312-
2313- """
2314- if type(data) != dict:
2315- logger.error("Cannot parse message data: %s", str(data))
2316- return {}
2317-
2318- n = {}
2319- if data.has_key("retweeted_status"):
2320- n["retweeted_by"] = self._user(data["user"] if "user" in data else data["sender"])
2321- if data.has_key("created_at"):
2322- n["time"] = parsetime(data["created_at"])
2323- data = data["retweeted_status"]
2324- else:
2325- n["retweeted_by"] = None
2326- if data.has_key("created_at"):
2327- n["time"] = parsetime(data["created_at"])
2328-
2329- m = self._common(data)
2330- for k in n:
2331- m[k] = n[k]
2332-
2333- if data.has_key("in_reply_to_status_id"):
2334- if data["in_reply_to_status_id"]:
2335- m["reply"] = {}
2336- m["reply"]["id"] = data["in_reply_to_status_id"]
2337- m["reply"]["nick"] = data["in_reply_to_screen_name"]
2338- if m["reply"]["id"] and m["reply"]["nick"]:
2339- m["reply"]["url"] = "/".join((self.account["url_prefix"], "notice", str(m["reply"]["id"])))
2340- else:
2341- m["reply"]["url"] = None
2342-
2343- m["sender"] = self._user(data["user"] if "user" in data else data["sender"])
2344- m["url"] = "/".join((self.account["url_prefix"], "notice", str(m["mid"])))
2345-
2346- return m
2347-
2348- def _private(self, data):
2349- """Sets the message type and privacy.
2350-
2351- Sets the message type and privacy if the
2352- message should be in the private stream.
2353-
2354- Also parses the recipient as both sent &
2355- received messages can be in the private stream.
2356- It sends the rest to _message() for further parsing
2357-
2358- Arguments:
2359- data -- A data object obtained from the StatusNet server containing a complete update
2360-
2361- Returns:
2362- m -- A data object compatible with inserting into the Gwibber database for that update
2363-
2364- """
2365- m = self._message(data)
2366- m["private"] = True
2367- m["recipient"] = {}
2368- m["recipient"]["name"] = data["recipient"]["name"]
2369- m["recipient"]["nick"] = data["recipient"]["screen_name"]
2370- m["recipient"]["id"] = data["recipient"]["id"]
2371- m["recipient"]["image"] = data["recipient"]["profile_image_url"]
2372- m["recipient"]["location"] = data["recipient"]["location"]
2373- m["recipient"]["url"] = "/".join((self.account["url_prefix"], m["recipient"]["nick"]))
2374- m["recipient"]["is_me"] = m["recipient"]["nick"].lower() == self.account["username"].lower()
2375- m["to_me"] = m["recipient"]["is_me"]
2376- return m
2377-
2378- def _result(self, data):
2379- """Called when a search is done in Gwibber.
2380-
2381- Parses the sender and sends the rest to _common()
2382- for further parsing.
2383-
2384- Args:
2385- data -- A data object obtained from the StatusNet server containing a complete update
2386-
2387- Returns:
2388- m -- A data object compatible with inserting into the Gwibber database for that update
2389-
2390- """
2391- m = self._common(data)
2392-
2393- if data["to_user_id"]:
2394- m["reply"] = {}
2395- m["reply"]["id"] = data["to_user_id"]
2396- m["reply"]["nick"] = data["to_user"]
2397-
2398- m["sender"] = {}
2399- m["sender"]["nick"] = data["from_user"]
2400- m["sender"]["id"] = data["from_user_id"]
2401- m["sender"]["image"] = data["profile_image_url"]
2402- m["sender"]["url"] = "/".join((self.account["url_prefix"], m["sender"]["nick"]))
2403- m["url"] = "/".join((self.account["url_prefix"], "notice", str(m["mid"])))
2404- return m
2405-
2406- def _profile(self, data):
2407- """Called when a user is clicked on.
2408-
2409- Arguments:
2410- data -- A data object obtained from the StatusNet server containing a complete user
2411-
2412- Returns:
2413- A data object compatible with inserting into the Gwibber database for that user.
2414-
2415- """
2416- return {
2417- "name": data.get("name", data["screen_name"]),
2418- "service": "statusnet",
2419- "stream": "profile",
2420- "account": self.account["id"],
2421- "mid": data["id"],
2422- "text": data.get("description", ""),
2423- "nick": data["screen_name"],
2424- "url": data.get("url", ""),
2425- "protected": data.get("protected", False),
2426- "statuses": data.get("statuses_count", 0),
2427- "followers": data.get("followers_count", 0),
2428- "friends": data.get("friends_count", 0),
2429- "following": data.get("following", 0),
2430- "favourites": data.get("favourites_count", 0),
2431- "image": data["profile_image_url"],
2432- "utc_offset": data.get("utc_offset", 0),
2433- "id": data["id"],
2434- "lang": data.get("lang", "en"),
2435- "verified": data.get("verified", False),
2436- "geo_enabled": data.get("geo_enabled", False),
2437- "time_zone": data.get("time_zone", "")
2438- }
2439-
2440- def _get(self, path, parse="message", post=False, single=False, **args):
2441- """Establishes a connection with the StatusNet server and gets the data requested.
2442-
2443- Requires authentication.
2444-
2445- Arguments:
2446- path -- The end of the URL to look up on the StatusNet server
2447- parse -- The function to use to parse the data returned (message by default)
2448- post -- True if using POST, for example the send operation. False if using GET, most operations other than send. (False by default)
2449- single -- True if a single checkin is requested, False if multiple (False by default)
2450- **args -- Arguments to be added to the URL when accessed
2451-
2452- Returns:
2453- A list of Gwibber compatible objects which have been parsed by the parse function.
2454-
2455- """
2456- if not self.account.has_key("access_token") and not self.account.has_key("secret_token"):
2457- logger.error("%s unexpected result - %s", PROTOCOL_INFO["name"], _("Account needs to be re-authorized"))
2458- return [{"error": {"type": "auth", "account": self.account, "message": _("Account needs to be re-authorized")}}]
2459-
2460- url = "/".join((self.account["url_prefix"], "api", path))
2461-
2462- self.sigmethod = oauth.OAuthSignatureMethod_HMAC_SHA1()
2463- self.consumer = oauth.OAuthConsumer("anonymous", "anonymous")
2464- self.token = oauth.OAuthToken(self.account["access_token"], self.account["secret_token"])
2465-
2466- parameters = util.compact(args)
2467- request = oauth.OAuthRequest.from_consumer_and_token(self.consumer, self.token,
2468- http_method=post and "POST" or "GET", http_url=url, parameters=parameters)
2469- request.sign_request(self.sigmethod, self.consumer, self.token)
2470-
2471- if post:
2472- data = network.Download(request.to_url(), parameters, post).get_json()
2473- else:
2474- data = network.Download(request.to_url(), None, post).get_json()
2475-
2476- resources.dump(self.account["service"], self.account["id"], data)
2477-
2478- if isinstance(data, dict) and data.get("error", 0):
2479- logger.error("%s failure - %s", PROTOCOL_INFO["name"], data["error"])
2480- if "authenticate" in data["error"]:
2481- return [{"error": {"type": "auth", "account": self.account, "message": data["error"]}}]
2482- else:
2483- return [{"error": {"type": "unknown", "account": self.account, "message": data["error"]}}]
2484- elif isinstance(data, str):
2485- logger.error("%s unexpected result - %s", PROTOCOL_INFO["name"], data)
2486- return [{"error": {"type": "unknown", "account": self.account, "message": data}}]
2487-
2488- if parse == "follow" or parse == "unfollow":
2489- if isinstance(data, dict) and data.get("error", 0):
2490- logstr = """%s: %s - %s""" % (PROTOCOL_INFO["name"], _("%s failed" % parse), data["error"])
2491- logger.error("%s", logstr)
2492- return [{"error": {"type": "auth", "account": self.account, "message": data["error"]}}]
2493- else:
2494- return [["friendships", {"type": parse, "account": self.account["id"], "service": self.account["service"],"user_id": data["id"], "nick": data["screen_name"]}]]
2495-
2496- if parse == "profile" and isinstance(data, dict):
2497- return self._profile(data)
2498-
2499- if single: return [getattr(self, "_%s" % parse)(data)]
2500- if parse: return [getattr(self, "_%s" % parse)(m) for m in data]
2501- else: return []
2502-
2503- return [self._result(m) for m in data]
2504-
2505- def _search(self, **args):
2506- """Establishes a connection with the StatusNet server and gets the results of a search.
2507-
2508- Does not require authentication
2509-
2510- Arguments:
2511- **args -- Example, if _search(tag=bar,name=foo) then **args is tag, name
2512-
2513- Returns:
2514- A list of Gwibber compatible objects which have been parsed by _result().
2515-
2516- """
2517- data = network.Download("%s/api/search.json" % self.account["url_prefix"], util.compact(args))
2518- data = data.get_json()
2519-
2520- if type(data) != dict:
2521- logger.error("Cannot parse search data: %s", str(data))
2522- return []
2523-
2524- return [self._result(m) for m in data["results"]]
2525-
2526- def __call__(self, opname, **args):
2527- return getattr(self, opname)(**args)
2528-
2529- def receive(self, count=util.COUNT, since=None):
2530- """Gets the latest updates and adds them to the database.
2531-
2532- Arguments:
2533- None
2534-
2535- Returns:
2536- A list of Gwibber compatible objects which have been parsed by _message().
2537-
2538- """
2539- return self._get("statuses/friends_timeline.json", count=count, since_id=since, source="Gwibber")
2540-
2541- def responses(self, count=util.COUNT, since=None):
2542- """Gets the latest replies and adds them to the database.
2543-
2544- Arguments:
2545- None
2546-
2547- Returns:
2548- A list of Gwibber compatible objects which have been parsed by _responses().
2549-
2550- """
2551- return self._get("statuses/mentions.json", count=count, since_id=since, source="Gwibber")
2552-
2553- def private(self, count=util.COUNT, since=None):
2554- """Gets the latest direct messages sent and received and adds them to the database.
2555-
2556- Arguments:
2557- None
2558-
2559- Returns:
2560- A list of Gwibber compatible objects which have been parsed by _private().
2561-
2562- """
2563- private = self._get("direct_messages.json", "private", count=count, since_id=since, source="Gwibber") or []
2564- private_sent = self._get("direct_messages/sent.json", "private", count=count, since_id=since, source="Gwibber") or []
2565- return private + private_sent
2566-
2567- def public(self, count=util.COUNT, since=None):
2568- """Gets the latest updates from the public timeline and adds them to the database.
2569-
2570- Arguments:
2571- None
2572-
2573- Returns:
2574- A list of Gwibber compatible objects which have been parsed by _message().
2575-
2576- """
2577- return self._get("statuses/public_timeline.json", source="Gwibber")
2578-
2579- def search(self, query, count=util.COUNT, since=None):
2580- """Gets the latest results from a search and adds them to the database.
2581-
2582- Arguments:
2583- None
2584-
2585- Returns:
2586- A list of Gwibber compatible objects which have been parsed by _search().
2587-
2588- """
2589- return self._search(q=query, rpp=count, since_id=since, source="Gwibber")
2590-
2591- def tag(self, query, count=util.COUNT, since=None):
2592- """Gets the latest results from a hashtag search and adds them to the database.
2593-
2594- Arguments:
2595- None
2596-
2597- Returns:
2598- A list of Gwibber compatible objects which have been parsed by _search()
2599-
2600- """
2601- return self._search(q="#%s" % query, count=count, since_id=since, source="Gwibber")
2602-
2603- def delete(self, message):
2604- """Deletes a specified update from the StatusNet server.
2605-
2606- Arguments:
2607- message -- A Gwibber compatible message object (from Gwibber's database)
2608-
2609- Returns:
2610- Nothing
2611-
2612- """
2613- self._get("statuses/destroy/%s.json" % message["mid"], None, post=True, do=1, source="Gwibber")
2614- return []
2615-
2616- def like(self, message):
2617- """Favorites a specified update on the StatusNet server.
2618-
2619- Arguments:
2620- message -- A Gwibber compatible message object (from Gwibber's database)
2621-
2622- Returns:
2623- Nothing
2624-
2625- """
2626- self._get("favorites/create/%s.json" % message["mid"], None, post=True, do=1, source="Gwibber")
2627- return []
2628-
2629- def send(self, message):
2630- """Sends a message to the StatusNet server.
2631-
2632- Arguments:
2633- message -- The update's text
2634-
2635- Returns:
2636- Nothing
2637-
2638- """
2639- return self._get("statuses/update.json", post=True, single=True, status=message, source="Gwibber")
2640-
2641- def send_private(self, message, private):
2642- """Sends a direct message to the StatusNet server.
2643-
2644- Arguments:
2645- message -- The update's text
2646- private -- A Gwibber compatible user object (from Gwibber's database)
2647-
2648- Returns:
2649- Nothing
2650-
2651- """
2652- return self._get("direct_messages/new.json", "private", post=True, single=True,
2653- text=message, screen_name=private["sender"]["nick"], source="Gwibber")
2654-
2655- def send_thread(self, message, target):
2656- """Sends a reply to a user on the StatusNet server.
2657-
2658- Arguments:
2659- message -- The update's text
2660- target -- A Gwibber compatible user object (from Gwibber's database)
2661-
2662- Returns:
2663- Nothing
2664-
2665- """
2666- return self._get("statuses/update.json", post=True, single=True,
2667- status=message, in_reply_to_status_id=target["mid"], source="Gwibber")
2668-
2669- def retweet(self, message):
2670- """Shares an update.
2671- Arguments:
2672- message -- A Gwibber compatible message object (from Gwibber's database)
2673-
2674- Returns:
2675- Nothing
2676-
2677- """
2678- return self._get("statuses/retweet/%s.json" % message["mid"], None, post=True, do=1, source="Gwibber")
2679-
2680- def follow(self, screen_name):
2681- """Follows a user.
2682-
2683- Arguments:
2684- screen_name -- The screen name of the user to be followed
2685-
2686- Returns:
2687- Nothing
2688-
2689- """
2690- return self._get("friendships/create.json", screen_name=screen_name, post=True, parse="follow", source="Gwibber")
2691-
2692- def unfollow(self, screen_name):
2693- """Unfollows a user.
2694-
2695- Arguments:
2696- screen_name -- The screen name of the user to be unfollowed
2697-
2698- Returns:
2699- Nothing
2700-
2701- """
2702- return self._get("friendships/destroy.json", screen_name=screen_name, post=True, parse="unfollow", source="Gwibber")
2703-
2704- def profile(self, id=None, count=None, since=None):
2705- """Gets a user's profile.
2706-
2707- Arguments:
2708- id -- The user's screen name
2709- count -- Number of updates to get
2710- since -- Time to get updates since
2711-
2712- Returns:
2713- A list of Gwibber compatible objects which have been parsed by _profile().
2714-
2715- """
2716- return self._get("users/show.json", screen_name=id, count=count, since_id=since, parse="profile", source="Gwibber")
2717-
2718- def user_messages(self, id=None, count=util.COUNT, since=None):
2719- """Gets a user's profile & timeline.
2720-
2721- Arguments:
2722- id -- The user's screen name
2723- count -- Number of updates to get
2724- since -- Time to get updates since
2725-
2726- Returns:
2727- A list of Gwibber compatible objects which have been parsed by _profile().
2728-
2729- """
2730- profiles = [self.profile(id)] or []
2731- messages = self._get("statuses/user_timeline.json", id=id, count=count, since_id=since, source="Gwibber") or []
2732- return messages + profiles

Subscribers

People subscribed via source and target branches