Merge lp:~barry/gwibber/bug-990145 into lp:gwibber

Proposed by Barry Warsaw
Status: Merged
Merged at revision: 1354
Proposed branch: lp:~barry/gwibber/bug-990145
Merge into: lp:gwibber
Diff against target: 697 lines (+286/-63)
16 files modified
README (+50/-0)
gwibber/microblog/dispatcher.py (+2/-2)
gwibber/microblog/plugins/facebook/__init__.py (+27/-8)
gwibber/microblog/plugins/flickr/__init__.py (+3/-6)
gwibber/microblog/plugins/friendfeed/__init__.py (+3/-2)
gwibber/microblog/plugins/identica/__init__.py (+7/-5)
gwibber/microblog/plugins/qaiku/__init__.py (+2/-1)
gwibber/microblog/plugins/statusnet/__init__.py (+7/-6)
gwibber/microblog/plugins/twitter/__init__.py (+7/-5)
gwibber/microblog/util/__init__.py (+3/-9)
gwibber/microblog/util/resources.py (+2/-2)
gwibber/time.py (+96/-0)
gwibber/util.py (+3/-3)
tests/plugins/test/__init__.py (+3/-5)
tests/python/unittests/test_time.py (+53/-0)
tests/python/utils/__init__.py (+18/-9)
To merge this branch: bzr merge lp:~barry/gwibber/bug-990145
Reviewer Review Type Date Requested Status
Ken VanDine Approve
Review via email: mp+103956@code.launchpad.net

Description of the change

It's a release goal for Quantal to ship only Python 3 on the desktop CD, so we
need to port Gwibber to Python 3. This branch doesn't do that. Instead, it
removes a dependency which we know will not be ported upstream any time soon.
I've heard from the mx project that they have no plans to port to Python 3.

This branch changes the uses of mx.DateTime to the built-in datetime module.
A 'make check' seems to pass locally, but it's entirely possible that I'm not
running the full suite correctly. I'm happy to make any additional changes
that might be necessary.

To post a comment you must log in.
Revision history for this message
Barry Warsaw (barry) wrote :

On May 10, 2012, at 06:09 PM, Ken VanDine wrote:

>Review: Needs Fixing
>
>Actually my last test didn't download any updates for facebook, so I purged
>all my facebook messages from the database. Now facebook refresh gives me
>this traceback:

Ah, so the FB data does have a timezone. Seems like a shallow bug, should be
easy to fix. I'll do that asap.

-Barry

>
>
>Dispatcher Thread-6 : ERROR <facebook:receive> Operation failed
>Dispatcher Thread-6 : DEBUG Traceback:
>Traceback (most recent call last):
> File "/home/ken/src/gwibber/trunk/gwibber/microblog/dispatcher.py", line 81, in run
> message_data = PROTOCOLS[account["service"]].Client(account)(opname, **args)
> File "/home/ken/src/gwibber/trunk/gwibber/microblog/plugins/facebook/__init__.py", line 276, in __call__
> return getattr(self, opname)(**args)
> File "/home/ken/src/gwibber/trunk/gwibber/microblog/plugins/facebook/__init__.py", line 292, in receive
> return [self._message(post) for post in data["data"]]
> File "/home/ken/src/gwibber/trunk/gwibber/microblog/plugins/facebook/__init__.py", line 151, in _message
> m["time"] = convert_time(data)
> File "/home/ken/src/gwibber/trunk/gwibber/microblog/plugins/facebook/__init__.py", line 80, in convert_time
> as_datetime = datetime.strptime(iso8601, ISO8601FORMAT)
> File "/usr/lib/python2.7/_strptime.py", line 328, in _strptime
> data_string[found.end():])
>ValueError: unconverted data remains: +0000
>

Revision history for this message
Ken VanDine (ken-vandine) wrote :

Thanks for the branch, review looks good and the unit tests pass. However it does fail with twitter, unfortunately we don't have tests that use real data from the services. It looks like it needs special handling like you did in facebook. Here is the traceback:

Dispatcher Thread-1 : ERROR <twitter:receive> Operation failed
Dispatcher Thread-1 : DEBUG Traceback:
Traceback (most recent call last):
  File "/home/ken/src/gwibber/trunk/gwibber/microblog/dispatcher.py", line 81, in run
    message_data = PROTOCOLS[account["service"]].Client(account)(opname, **args)
  File "/home/ken/src/gwibber/trunk/gwibber/microblog/plugins/twitter/__init__.py", line 411, in __call__
    return getattr(self, opname)(**args)
  File "/home/ken/src/gwibber/trunk/gwibber/microblog/plugins/twitter/__init__.py", line 414, in receive
    return self._get("statuses/home_timeline.json", include_entities=1, count=count, since_id=since)
  File "/home/ken/src/gwibber/trunk/gwibber/microblog/plugins/twitter/__init__.py", line 397, in _get
    if parse: return [getattr(self, "_%s" % parse)(m) for m in data]
  File "/home/ken/src/gwibber/trunk/gwibber/microblog/plugins/twitter/__init__.py", line 235, in _message
    n["time"] = util.parsetime(data["created_at"])
  File "/home/ken/src/gwibber/trunk/gwibber/microblog/util/__init__.py", line 36, in parsetime
    dt = datetime.strptime(t, ISO8601FORMAT)
  File "/usr/lib/python2.7/_strptime.py", line 325, in _strptime
    (data_string, format))
ValueError: time data 'Thu May 10 17:52:38 +0000 2012' does not match format '%Y-%m-%dT%H:%M:%S'

review: Needs Fixing
Revision history for this message
Ken VanDine (ken-vandine) wrote :

Actually my last test didn't download any updates for facebook, so I purged all my facebook messages from the database. Now facebook refresh gives me this traceback:

Dispatcher Thread-6 : ERROR <facebook:receive> Operation failed
Dispatcher Thread-6 : DEBUG Traceback:
Traceback (most recent call last):
  File "/home/ken/src/gwibber/trunk/gwibber/microblog/dispatcher.py", line 81, in run
    message_data = PROTOCOLS[account["service"]].Client(account)(opname, **args)
  File "/home/ken/src/gwibber/trunk/gwibber/microblog/plugins/facebook/__init__.py", line 276, in __call__
    return getattr(self, opname)(**args)
  File "/home/ken/src/gwibber/trunk/gwibber/microblog/plugins/facebook/__init__.py", line 292, in receive
    return [self._message(post) for post in data["data"]]
  File "/home/ken/src/gwibber/trunk/gwibber/microblog/plugins/facebook/__init__.py", line 151, in _message
    m["time"] = convert_time(data)
  File "/home/ken/src/gwibber/trunk/gwibber/microblog/plugins/facebook/__init__.py", line 80, in convert_time
    as_datetime = datetime.strptime(iso8601, ISO8601FORMAT)
  File "/usr/lib/python2.7/_strptime.py", line 328, in _strptime
    data_string[found.end():])
ValueError: unconverted data remains: +0000

review: Needs Fixing
lp:~barry/gwibber/bug-990145 updated
1345. By Barry Warsaw

 - Add the most minimal of unittests, with instructions for running them in
   the README file.

 - Add unittests for parsetime().

 - Support more formats in parsetime(), e.g. timezone aware strings (but only
   for UTC, i.e. +0000), and also for alternative ISO 8601 (space instead of
   'T').

Revision history for this message
Barry Warsaw (barry) wrote :

I've pushed an update which should fix the parsing in the last comment. I also added the most minimal of unittests, based on our out-of-band discussion. See the README for details. It's ugly and not tied into `make check` but at least it's a start. ;)

lp:~barry/gwibber/bug-990145 updated
1346. By Barry Warsaw

trunk merge

Revision history for this message
Barry Warsaw (barry) wrote :

I think the last update may not completely fix this problem. I'm working on an update.

lp:~barry/gwibber/bug-990145 updated
1347. By Barry Warsaw

 - Remove some unused imports, and other import cleanups.
 - Avoid circular imports by moving parsetime() to new module gwibber.time
 - Refactor convert_time() to use parsetime() in order to fix the bugs
   everywhere.

Revision history for this message
Barry Warsaw (barry) wrote :

Latest update pushed, which should work (seems to locally anyway ;). You'll see the diff got bigger because I had to fiddle with the imports to prevent some circular import problems.

lp:~barry/gwibber/bug-990145 updated
1348. By Barry Warsaw

 - Handle Twitter and Facebook nonconformity to standards.
 - Refactor the code to be more readable.
 - Improve the comments.

Revision history for this message
Ken VanDine (ken-vandine) wrote :

twitter and facebook confirmed to work, however I tested with identica and status.net and they aren't using a UTC time and give the same ValueError. Here is an example time string:

Wed Mar 21 02:30:31 -0400 2012

There is also a typo in the flickr plugin:
- timetup = datetime.utcfromtimestap(int(t)).timetuple()
+ timetup = datetime.utcfromtimestamp(int(t)).timetuple()

The good news is that will be the last of the supported plugins, so once identica and statusnet are working this is good to go!

Thanks!

review: Needs Fixing
Revision history for this message
Barry Warsaw (barry) wrote :

On May 11, 2012, at 11:42 PM, Ken VanDine wrote:

>Review: Needs Fixing
>
>twitter and facebook confirmed to work, however I tested with identica and
>status.net and they aren't using a UTC time and give the same ValueError.
>Here is an example time string:
>
>Wed Mar 21 02:30:31 -0400 2012

Madness!

What does Gwibber expect us to do with non-UTC timezones? Just dropping the
tz doesn't seem right. OTOH, we have to convert them to naive datetimes. So
I'm guessing we parse them into non-UTC tz-aware datetimes, then convert them
to UTC, then make them tz-naive. Does that sound right?

>There is also a typo in the flickr plugin:
>- timetup = datetime.utcfromtimestap(int(t)).timetuple()
>+ timetup = datetime.utcfromtimestamp(int(t)).timetuple()
>
>The good news is that will be the last of the supported plugins, so once
>identica and statusnet are working this is good to go!

Yay! :)

Cheers,
-Barry

Revision history for this message
Ken VanDine (ken-vandine) wrote :

On Fri, 2012-05-11 at 16:52 -0700, Barry Warsaw wrote:
> On May 11, 2012, at 11:42 PM, Ken VanDine wrote:
>
> >Review: Needs Fixing
> >
> >twitter and facebook confirmed to work, however I tested with identica and
> >status.net and they aren't using a UTC time and give the same ValueError.
> >Here is an example time string:
> >
> >Wed Mar 21 02:30:31 -0400 2012
>
> Madness!
>
> What does Gwibber expect us to do with non-UTC timezones? Just dropping the
> tz doesn't seem right. OTOH, we have to convert them to naive datetimes. So
> I'm guessing we parse them into non-UTC tz-aware datetimes, then convert them
> to UTC, then make them tz-naive. Does that sound right?

Yeah, I think that is the only thing we can do. Sure wish they were
consistent :(

Thanks!

>
> >There is also a typo in the flickr plugin:
> >- timetup = datetime.utcfromtimestap(int(t)).timetuple()
> >+ timetup = datetime.utcfromtimestamp(int(t)).timetuple()
> >
> >The good news is that will be the last of the supported plugins, so once
> >identica and statusnet are working this is good to go!
>
> Yay! :)
>
> Cheers,
> -Barry
>

--
Ken VanDine
Ubuntu Desktop Integration Engineer
Canonical, Ltd.

lp:~barry/gwibber/bug-990145 updated
1349. By Barry Warsaw

 - Change the flickr plugin to use the parsetime() utility.
 - Remove some unused imports from the flickr plugin.
 - parsetime(): Elaborate so that non-UTC timezones are handled properly,
   which is used by identica and status.net. We have to use regexps here
   though since %z isn't universally supported until Python 3.

Revision history for this message
Barry Warsaw (barry) wrote :

On May 11, 2012, at 11:42 PM, Ken VanDine wrote:

>twitter and facebook confirmed to work, however I tested with identica and
>status.net and they aren't using a UTC time and give the same ValueError.
>Here is an example time string:
>
>Wed Mar 21 02:30:31 -0400 2012

Okay, I think I'm handling this case now. Manual timezone math is always fun
(you have to subtract the value from your local time to get you to UTC).
Please do check my math! (And the test cases. :)

>There is also a typo in the flickr plugin:
>- timetup = datetime.utcfromtimestap(int(t)).timetuple()
>+ timetup = datetime.utcfromtimestamp(int(t)).timetuple()

I changed this to use gwibber.time.parsetime().

>The good news is that will be the last of the supported plugins, so once
>identica and statusnet are working this is good to go!

Great! Hopefully this time <wink> will do it.

Update pushed.

Revision history for this message
Ken VanDine (ken-vandine) wrote :

Looks great, thanks!

review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== modified file 'README'
--- README 2012-05-10 17:20:35 +0000
+++ README 2012-05-12 02:14:17 +0000
@@ -0,0 +1,50 @@
1Installing Gwibber
2==================
3
4 Requirements
5 ------------
6
7 Please note that the version numbers listed below only reflect my test
8 environment. Gwibber is known to work with those specific versions, but
9 will probably work fine on most current desktop distributions that include
10 Python's WebKit GTK+ bindings.
11
12 * python (2.5)
13 * python-dbus (0.80.2)
14 * python-gtk2 (2.10.4)
15 * python-gconf (2.18.0)
16 * python-imaging (1.1.6)
17 * python-notify (0.1.1)
18 * python-webkitgtk (1.0.1)
19 * python-simplejson (1.9.1)
20 * python-distutils-extra
21 * python-feedparser (4.1)
22 * python-xdg (0.15)
23 * python-mako (0.2.2)
24 * python-pycurl
25
26 Installation
27 ------------
28
29 Gwibber uses Python's distutils framework for installation. In order to
30 install Gwibber, you will need root access. To install Gwibber, perform
31 the following command as root:
32
33 $ python setup.py install
34
35 Run Gwibber
36 -----------
37
38 If you installed Gwibber using the setup.py script, you can launch the
39 program by typing "gwibber" at the command line. If you want to run
40 Gwibber without installing it, start "bin/gwibber" from within the
41 Gwibber directory.
42
43 Testing
44 -------
45
46 You can run the dbus isolated tests by cd'ing into the tests directory
47 and running `make check`.
48
49 You can run the Python unittests by cd'ing into tests/python/unittests
50 and running `PYTHONPATH=../../.. python -m unittest discover -v`
051
=== modified file 'gwibber/microblog/dispatcher.py'
--- gwibber/microblog/dispatcher.py 2012-03-27 15:48:44 +0000
+++ gwibber/microblog/dispatcher.py 2012-05-12 02:14:17 +0000
@@ -3,7 +3,7 @@
33
4import traceback, json, random, string4import traceback, json, random, string
5import dbus, dbus.service5import dbus, dbus.service
6import sqlite3, mx.DateTime, re, uuid6import sqlite3, re, uuid
7import urlshorter, storage, network, util, uploader7import urlshorter, storage, network, util, uploader
8from gettext import lgettext as _8from gettext import lgettext as _
99
@@ -943,7 +943,7 @@
943 operations.append(o)943 operations.append(o)
944 944
945 if operations:945 if operations:
946 logger.debug("** Starting Refresh - %s **", mx.DateTime.now())946 logger.debug("** Starting Refresh - %s **", datetime.now())
947 self.LoadingStarted()947 self.LoadingStarted()
948 self.perform_async_operation(operations)948 self.perform_async_operation(operations)
949949
950950
=== modified file 'gwibber/microblog/plugins/facebook/__init__.py'
--- gwibber/microblog/plugins/facebook/__init__.py 2012-03-09 19:34:33 +0000
+++ gwibber/microblog/plugins/facebook/__init__.py 2012-05-12 02:14:17 +0000
@@ -2,8 +2,7 @@
22
3from gwibber.microblog import network, util3from gwibber.microblog import network, util
4from gwibber.microblog.util import resources4from gwibber.microblog.util import resources
5import hashlib, mx.DateTime, time5import hashlib, time
6from os.path import join, getmtime, exists
7from gettext import lgettext as _6from gettext import lgettext as _
8from gwibber.microblog.util.const import *7from gwibber.microblog.util.const import *
9# Try to import * from custom, install custom.py to include packaging 8# Try to import * from custom, install custom.py to include packaging
@@ -12,11 +11,15 @@
12 from gwibber.microblog.util.custom import *11 from gwibber.microblog.util.custom import *
13except:12except:
14 pass13 pass
14from gwibber.time import parsetime
15
16from datetime import datetime, timedelta
1517
16import logging18import logging
17logger = logging.getLogger("Facebook")19logger = logging.getLogger("Facebook")
18logger.debug("Initializing.")20logger.debug("Initializing.")
1921
22
20PROTOCOL_INFO = {23PROTOCOL_INFO = {
21 "name": "Facebook",24 "name": "Facebook",
22 "version": "1.1",25 "version": "1.1",
@@ -56,6 +59,21 @@
56URL_PREFIX = "https://graph.facebook.com/"59URL_PREFIX = "https://graph.facebook.com/"
57POST_URL = "http://www.facebook.com/profile.php?id=%s&v=feed&story_fbid=%s&ref=mf"60POST_URL = "http://www.facebook.com/profile.php?id=%s&v=feed&story_fbid=%s&ref=mf"
5861
62
63def convert_time(data):
64 """Extract and convert the time to a timestamp (seconds since Epoch).
65
66 First, look for the 'updated_time' key in the data, falling back to
67 'created_time' if the former is missing.
68 """
69 # Convert from an ISO 8601 format time string to a Epoch timestamp.
70 # Assume standard ISO 8601 'T' separator, no microseconds, and naive
71 # time zone as per:
72 # https://developers.facebook.com/docs/reference/api/event/
73 time_string = data.get('updated_time', data['created_time'])
74 return parsetime(time_string)
75
76
59class Client:77class Client:
60 def __init__(self, acct):78 def __init__(self, acct):
61 self.account = acct79 self.account = acct
@@ -122,7 +140,7 @@
122 m["service"] = "facebook"140 m["service"] = "facebook"
123 m["account"] = self.account["id"]141 m["account"] = self.account["id"]
124142
125 m["time"] = int(mx.DateTime.DateTimeFrom(str(data.get("updated_time", data["created_time"]))))143 m["time"] = convert_time(data)
126 m["url"] = "https://facebook.com/" + data["id"].split("_")[0] + "/posts/" + data["id"].split("_")[1]144 m["url"] = "https://facebook.com/" + data["id"].split("_")[0] + "/posts/" + data["id"].split("_")[1]
127 145
128146
@@ -233,7 +251,7 @@
233 for item in data["comments"]["data"]:251 for item in data["comments"]["data"]:
234 m["comments"].append({252 m["comments"].append({
235 "text": item["message"],253 "text": item["message"],
236 "time": int(mx.DateTime.DateTimeFrom(str(data.get("updated_time", data["created_time"])))),254 "time": convert_time(data),
237 "sender": self._sender(item["from"]),255 "sender": self._sender(item["from"]),
238 })256 })
239257
@@ -250,15 +268,16 @@
250 return getattr(self, opname)(**args)268 return getattr(self, opname)(**args)
251269
252 def receive(self, since=None):270 def receive(self, since=None):
253 if not since:271 if since is None:
254 since = int(mx.DateTime.DateTimeFromTicks(mx.DateTime.localtime()) - mx.DateTime.TimeDelta(hours=240.0))272 past = datetime.now() - timedelta(hours=240.0)
255 else:273 else:
256 since = int(mx.DateTime.DateTimeFromTicks(since).localtime())274 past = datetime.fromtimestamp(since)
257275
276 since = int(time.mktime(past.timetuple()))
258 data = self._get("me/home", since=since, limit=100)277 data = self._get("me/home", since=since, limit=100)
259 278
260 logger.debug("<STATS> facebook:receive account:%s since:%s size:%s",279 logger.debug("<STATS> facebook:receive account:%s since:%s size:%s",
261 self.account["id"], mx.DateTime.DateTimeFromTicks(since), len(str(data)))280 self.account["id"], datetime.fromtimestamp(since), len(str(data)))
262 281
263 if not self._check_error(data):282 if not self._check_error(data):
264 try:283 try:
265284
=== modified file 'gwibber/microblog/plugins/flickr/__init__.py'
--- gwibber/microblog/plugins/flickr/__init__.py 2012-02-10 10:06:59 +0000
+++ gwibber/microblog/plugins/flickr/__init__.py 2012-05-12 02:14:17 +0000
@@ -1,11 +1,10 @@
1from gwibber.microblog import network, util1from gwibber.microblog import network
2from gwibber.microblog.util import resources2from gwibber.time import parsetime
33
4import logging4import logging
5logger = logging.getLogger("Flickr")5logger = logging.getLogger("Flickr")
6logger.debug("Initializing.")6logger.debug("Initializing.")
77
8import re, mx.DateTime
9from gettext import lgettext as _8from gettext import lgettext as _
109
11PROTOCOL_INFO = {10PROTOCOL_INFO = {
@@ -37,8 +36,6 @@
37IMAGE_URL = "http://farm%s.static.flickr.com/%s/%s_%s_%s.jpg"36IMAGE_URL = "http://farm%s.static.flickr.com/%s/%s_%s_%s.jpg"
38IMAGE_PAGE_URL = "http://www.flickr.com/photos/%s/%s"37IMAGE_PAGE_URL = "http://www.flickr.com/photos/%s/%s"
3938
40def parse_time(t):
41 return mx.DateTime.DateTimeFromTicks(int(t)).gmtime().ticks()
4239
43class Client:40class Client:
44 def __init__(self, acct):41 def __init__(self, acct):
@@ -49,7 +46,7 @@
49 m["mid"] = str(data["id"])46 m["mid"] = str(data["id"])
50 m["service"] = "flickr"47 m["service"] = "flickr"
51 m["account"] = self.account["id"]48 m["account"] = self.account["id"]
52 m["time"] = parse_time(data["dateupload"])49 m["time"] = parsetime(data["dateupload"])
53 m["source"] = False50 m["source"] = False
54 m["text"] = data["title"]51 m["text"] = data["title"]
5552
5653
=== modified file 'gwibber/microblog/plugins/friendfeed/__init__.py'
--- gwibber/microblog/plugins/friendfeed/__init__.py 2012-02-10 10:06:59 +0000
+++ gwibber/microblog/plugins/friendfeed/__init__.py 2012-05-12 02:14:17 +0000
@@ -1,5 +1,6 @@
1from gwibber.microblog import network, util1from gwibber.microblog import network, util
2from gwibber.microblog.util import resources2from gwibber.microblog.util import resources
3from gwibber.time import parsetime
34
4import logging5import logging
5logger = logging.getLogger("FriendFeed")6logger = logging.getLogger("FriendFeed")
@@ -61,7 +62,7 @@
61 "mid": data["id"],62 "mid": data["id"],
62 "service": "friendfeed",63 "service": "friendfeed",
63 "account": self.account["id"],64 "account": self.account["id"],
64 "time": util.parsetime(data["published"]),65 "time": parsetime(data["published"]),
65 "source": data.get("via", {}).get("name", None),66 "source": data.get("via", {}).get("name", None),
66 "text": data["title"],67 "text": data["title"],
67 "html": util.linkify(data["title"]),68 "html": util.linkify(data["title"]),
@@ -86,7 +87,7 @@
86 for item in data["comments"][-3:]:87 for item in data["comments"][-3:]:
87 m["comments"].append({88 m["comments"].append({
88 "text": item["body"],89 "text": item["body"],
89 "time": util.parsetime(item["date"]),90 "time": parsetime(item["date"]),
90 "sender": self._sender(item["user"]),91 "sender": self._sender(item["user"]),
91 })92 })
9293
9394
=== modified file 'gwibber/microblog/plugins/identica/__init__.py'
--- gwibber/microblog/plugins/identica/__init__.py 2012-02-13 20:39:02 +0000
+++ gwibber/microblog/plugins/identica/__init__.py 2012-05-12 02:14:17 +0000
@@ -1,7 +1,9 @@
1from gettext import lgettext as _
2from oauth import oauth
3
1from gwibber.microblog import network, util4from gwibber.microblog import network, util
2from oauth import oauth
3from gwibber.microblog.util import resources5from gwibber.microblog.util import resources
4from gettext import lgettext as _6from gwibber.time import parsetime
57
6import logging8import logging
7logger = logging.getLogger("Identica")9logger = logging.getLogger("Identica")
@@ -71,7 +73,7 @@
71 m["service"] = "identica"73 m["service"] = "identica"
72 m["account"] = self.account["id"]74 m["account"] = self.account["id"]
73 if data.has_key("created_at"):75 if data.has_key("created_at"):
74 m["time"] = util.parsetime(data["created_at"])76 m["time"] = parsetime(data["created_at"])
75 m["source"] = data.get("source", False)77 m["source"] = data.get("source", False)
76 m["text"] = util.unescape(data["text"])78 m["text"] = util.unescape(data["text"])
77 m["to_me"] = ("@%s" % self.account["username"]) in data["text"]79 m["to_me"] = ("@%s" % self.account["username"]) in data["text"]
@@ -130,12 +132,12 @@
130 if data.has_key("retweeted_status"):132 if data.has_key("retweeted_status"):
131 n["retweeted_by"] = self._user(data["user"] if "user" in data else data["sender"])133 n["retweeted_by"] = self._user(data["user"] if "user" in data else data["sender"])
132 if data.has_key("created_at"):134 if data.has_key("created_at"):
133 n["time"] = util.parsetime(data["created_at"])135 n["time"] = parsetime(data["created_at"])
134 data = data["retweeted_status"]136 data = data["retweeted_status"]
135 else:137 else:
136 n["retweeted_by"] = None138 n["retweeted_by"] = None
137 if data.has_key("created_at"):139 if data.has_key("created_at"):
138 n["time"] = util.parsetime(data["created_at"])140 n["time"] = parsetime(data["created_at"])
139141
140 m = self._common(data)142 m = self._common(data)
141143
142144
=== modified file 'gwibber/microblog/plugins/qaiku/__init__.py'
--- gwibber/microblog/plugins/qaiku/__init__.py 2012-02-10 10:06:59 +0000
+++ gwibber/microblog/plugins/qaiku/__init__.py 2012-05-12 02:14:17 +0000
@@ -1,5 +1,6 @@
1from gwibber.microblog import network, util1from gwibber.microblog import network, util
2from gwibber.microblog.util import resources2from gwibber.microblog.util import resources
3from gwibber.time import parsetime
34
4import logging5import logging
5logger = logging.getLogger("Qaiku")6logger = logging.getLogger("Qaiku")
@@ -46,7 +47,7 @@
46 m["mid"] = str(data["id"])47 m["mid"] = str(data["id"])
47 m["service"] = "qaiku"48 m["service"] = "qaiku"
48 m["account"] = self.account["id"]49 m["account"] = self.account["id"]
49 m["time"] = util.parsetime(data["created_at"])50 m["time"] = parsetime(data["created_at"])
50 m["text"] = data["text"]51 m["text"] = data["text"]
51 m["to_me"] = ("@%s" % self.account["username"]) in data["text"]52 m["to_me"] = ("@%s" % self.account["username"]) in data["text"]
5253
5354
=== modified file 'gwibber/microblog/plugins/statusnet/__init__.py'
--- gwibber/microblog/plugins/statusnet/__init__.py 2012-02-13 20:39:02 +0000
+++ gwibber/microblog/plugins/statusnet/__init__.py 2012-05-12 02:14:17 +0000
@@ -1,8 +1,9 @@
1import re1from gettext import lgettext as _
2from oauth import oauth
3
2from gwibber.microblog import network, util4from gwibber.microblog import network, util
3from oauth import oauth
4from gwibber.microblog.util import resources5from gwibber.microblog.util import resources
5from gettext import lgettext as _6from gwibber.time import parsetime
67
7import logging8import logging
8logger = logging.getLogger("StatusNet")9logger = logging.getLogger("StatusNet")
@@ -71,7 +72,7 @@
71 m["service"] = "statusnet"72 m["service"] = "statusnet"
72 m["account"] = self.account["id"]73 m["account"] = self.account["id"]
73 if data.has_key("created_at"):74 if data.has_key("created_at"):
74 m["time"] = util.parsetime(data["created_at"])75 m["time"] = parsetime(data["created_at"])
75 m["source"] = data.get("source", False)76 m["source"] = data.get("source", False)
76 m["text"] = util.unescape(data["text"])77 m["text"] = util.unescape(data["text"])
77 m["to_me"] = ("@%s" % self.account["username"]) in data["text"]78 m["to_me"] = ("@%s" % self.account["username"]) in data["text"]
@@ -130,12 +131,12 @@
130 if data.has_key("retweeted_status"):131 if data.has_key("retweeted_status"):
131 n["retweeted_by"] = self._user(data["user"] if "user" in data else data["sender"])132 n["retweeted_by"] = self._user(data["user"] if "user" in data else data["sender"])
132 if data.has_key("created_at"):133 if data.has_key("created_at"):
133 n["time"] = util.parsetime(data["created_at"])134 n["time"] = parsetime(data["created_at"])
134 data = data["retweeted_status"]135 data = data["retweeted_status"]
135 else:136 else:
136 n["retweeted_by"] = None137 n["retweeted_by"] = None
137 if data.has_key("created_at"):138 if data.has_key("created_at"):
138 n["time"] = util.parsetime(data["created_at"])139 n["time"] = parsetime(data["created_at"])
139140
140 m = self._common(data)141 m = self._common(data)
141 for k in n:142 for k in n:
142143
=== modified file 'gwibber/microblog/plugins/twitter/__init__.py'
--- gwibber/microblog/plugins/twitter/__init__.py 2012-03-30 14:32:20 +0000
+++ gwibber/microblog/plugins/twitter/__init__.py 2012-05-12 02:14:17 +0000
@@ -1,8 +1,10 @@
1from gwibber.microblog import network, util
2import cgi1import cgi
2from gettext import lgettext as _
3from oauth import oauth3from oauth import oauth
4
5from gwibber.microblog import network, util
4from gwibber.microblog.util import resources6from gwibber.microblog.util import resources
5from gettext import lgettext as _7from gwibber.time import parsetime
68
7import logging9import logging
8logger = logging.getLogger("Twitter")10logger = logging.getLogger("Twitter")
@@ -79,7 +81,7 @@
79 m["service"] = "twitter"81 m["service"] = "twitter"
80 m["account"] = self.account["id"]82 m["account"] = self.account["id"]
81 if data.has_key("created_at"):83 if data.has_key("created_at"):
82 m["time"] = util.parsetime(data["created_at"])84 m["time"] = parsetime(data["created_at"])
83 m["text"] = util.unescape(data["text"])85 m["text"] = util.unescape(data["text"])
84 m["text"] = cgi.escape(m["text"])86 m["text"] = cgi.escape(m["text"])
85 m["content"] = m["text"]87 m["content"] = m["text"]
@@ -227,12 +229,12 @@
227 if data.has_key("retweeted_status"):229 if data.has_key("retweeted_status"):
228 n["retweeted_by"] = self._user(data["user"] if "user" in data else data["sender"])230 n["retweeted_by"] = self._user(data["user"] if "user" in data else data["sender"])
229 if data.has_key("created_at"):231 if data.has_key("created_at"):
230 n["time"] = util.parsetime(data["created_at"])232 n["time"] = parsetime(data["created_at"])
231 data = data["retweeted_status"]233 data = data["retweeted_status"]
232 else:234 else:
233 n["retweeted_by"] = None235 n["retweeted_by"] = None
234 if data.has_key("created_at"):236 if data.has_key("created_at"):
235 n["time"] = util.parsetime(data["created_at"])237 n["time"] = parsetime(data["created_at"])
236238
237 m = self._common(data)239 m = self._common(data)
238 for k in n:240 for k in n:
239241
=== modified file 'gwibber/microblog/util/__init__.py'
--- gwibber/microblog/util/__init__.py 2012-04-20 21:16:37 +0000
+++ gwibber/microblog/util/__init__.py 2012-05-12 02:14:17 +0000
@@ -1,7 +1,7 @@
1import os, locale, re, mx.DateTime, cgi, httplib21import re, cgi, httplib2
2import resources2from gwibber.microblog.util import resources
3import dbus3import dbus
4from const import *4from gwibber.microblog.util.const import *
5from htmlentitydefs import name2codepoint5from htmlentitydefs import name2codepoint
66
7import logging7import logging
@@ -16,12 +16,6 @@
1616
17COUNT = 20017COUNT = 200
1818
19def parsetime(t):
20 locale.setlocale(locale.LC_TIME, 'C')
21 result = mx.DateTime.Parser.DateTimeFromString(t)
22 locale.setlocale(locale.LC_TIME, '')
23 return result.ticks()
24
25URL_SCHEMES = ('http', 'https', 'ftp', 'mailto', 'news', 'gopher',19URL_SCHEMES = ('http', 'https', 'ftp', 'mailto', 'news', 'gopher',
26 'nntp', 'telnet', 'wais', 'prospero', 'aim', 'webcal')20 'nntp', 'telnet', 'wais', 'prospero', 'aim', 'webcal')
2721
2822
=== modified file 'gwibber/microblog/util/resources.py'
--- gwibber/microblog/util/resources.py 2012-03-20 16:31:03 +0000
+++ gwibber/microblog/util/resources.py 2012-05-12 02:14:17 +0000
@@ -9,7 +9,7 @@
9from os import makedirs, remove, environ9from os import makedirs, remove, environ
10from os.path import join, isdir, realpath, exists10from os.path import join, isdir, realpath, exists
11import Image11import Image
12import mx.DateTime12from datetime import datetime
13from gwibber.microblog import network13from gwibber.microblog import network
14from gwibber.microblog.util.const import *14from gwibber.microblog.util.const import *
15import inspect15import inspect
@@ -211,7 +211,7 @@
211 dump_cache_dir = realpath(join(CACHE_BASE_DIR, "gwibber", "dump", service))211 dump_cache_dir = realpath(join(CACHE_BASE_DIR, "gwibber", "dump", service))
212 if not isdir(dump_cache_dir):212 if not isdir(dump_cache_dir):
213 makedirs(dump_cache_dir)213 makedirs(dump_cache_dir)
214 dump_cache_file = join(dump_cache_dir, (aid + "." + str(mx.DateTime.now()) + "." + operation))214 dump_cache_file = join(dump_cache_dir, (aid + "." + str(datetime.now()) + "." + operation))
215215
216 if not exists(dump_cache_file) or len(open(dump_cache_file, "r").read()) < 1:216 if not exists(dump_cache_file) or len(open(dump_cache_file, "r").read()) < 1:
217 logger.debug("Dumping test data %s - %s - %s", service, aid, operation)217 logger.debug("Dumping test data %s - %s - %s", service, aid, operation)
218218
=== added file 'gwibber/time.py'
--- gwibber/time.py 1970-01-01 00:00:00 +0000
+++ gwibber/time.py 2012-05-12 02:14:17 +0000
@@ -0,0 +1,96 @@
1from __future__ import absolute_import
2
3import re
4import time
5import locale
6
7from contextlib import contextmanager
8from datetime import datetime, timedelta
9
10
11# Date time formats. Assume no microseconds and no timezone.
12ISO8601_FORMAT = '%Y-%m-%dT%H:%M:%S'
13TWITTER_FORMAT = '%a %b %d %H:%M:%S %Y'
14
15
16@contextmanager
17def c_locale():
18 locale.setlocale(locale.LC_TIME, 'C')
19 try:
20 yield
21 finally:
22 locale.setlocale(locale.LC_TIME, '')
23
24
25def iso8601(t):
26 return datetime.strptime(t, ISO8601_FORMAT)
27
28def iso8601alt(t):
29 return datetime.strptime(t, ISO8601_FORMAT.replace('T', ' '))
30
31def twitter(t):
32 return datetime.strptime(t, TWITTER_FORMAT)
33
34
35def parsetime(t):
36 """Parse an ISO 8601 datetime string and return seconds since epoch.
37
38 This accepts either a naive (i.e. timezone-less) string or a timezone
39 aware string. The timezone must start with a + or - and must be followed
40 by exactly four digits. This string is parsed and converted to UTC. This
41 value is then converted to an integer seconds since epoch.
42 """
43 with c_locale():
44 # In order to parse the UTC timezone (e.g. +0000), you'd think we
45 # could just append %z on the format, but that doesn't work
46 # across Python versions. On some platforms in Python 2.7, %z is not
47 # a valid format directive, and its use will raise a ValueError.
48 #
49 # In Python 3.2, strptime() is implemented in Python, so it *is*
50 # supported everywhere, but we still can't rely on it because of the
51 # non-ISO 8601 formats that some APIs use (I'm looking at you Twitter
52 # and Facebook). We'll use a regular expression to tear out the
53 # timezone string and do the conversion ourselves.
54 #
55 # tz_offset is a list for two reasons. First, so we can play games
56 # with scoped access to the variable inside capture_tz() without the
57 # use of `nonlocal` which doesn't exist in Python 2. Also, we'll use
58 # this as a cheap way of ensuring there aren't multiple matching
59 # timezone strings in a single input string.
60 tz_offset = []
61 def capture_tz(match_object):
62 tz_string = match_object.group('tz')
63 if tz_string is not None:
64 # It's possible that we'll see more than one substring
65 # matching the timezone pattern. It should be highly unlikely
66 # so we won't test for that here, at least not now.
67 #
68 # The tz_offset is positive, so it must be subtracted from the
69 # naive datetime in order to return it to UTC. E.g.
70 # 13:00 -0400 is 17:00 +0000
71 # or
72 # 1300 - (-0400 / 100)
73 tz_offset.append(timedelta(hours=int(tz_string) / 100))
74 # Return the empty string so as to remove the timezone pattern
75 # from the string we're going to parse.
76 return ''
77 naive_t = re.sub(r'[ ]*(?P<tz>[-+]\d{4})', capture_tz, t)
78 if len(tz_offset) == 0:
79 tz_offset = timedelta()
80 elif len(tz_offset) == 1:
81 tz_offset = tz_offset[0]
82 else:
83 raise ValueError('Unsupported time string: {0}'.format(t))
84 for parser in (iso8601, iso8601alt, twitter):
85 try:
86 dt = parser(naive_t) - tz_offset
87 except ValueError:
88 pass
89 else:
90 break
91 else:
92 # Nothing matched.
93 raise ValueError('Unsupported time string: {0}'.format(t))
94 # We must have gotten a valid datetime. Convert it to Epoch seconds.
95 timetup = dt.timetuple()
96 return int(time.mktime(timetup))
097
=== modified file 'gwibber/util.py'
--- gwibber/util.py 2012-02-13 19:59:49 +0000
+++ gwibber/util.py 2012-05-12 02:14:17 +0000
@@ -1,4 +1,4 @@
1import dbus, os, mx.DateTime, webbrowser1import dbus, os, webbrowser
2import subprocess2import subprocess
3from gi.repository import GLib, Gdk, GdkPixbuf, Gtk3from gi.repository import GLib, Gdk, GdkPixbuf, Gtk
4from microblog.util import resources4from microblog.util import resources
@@ -105,8 +105,8 @@
105105
106def generate_time_string(t):106def generate_time_string(t):
107 if isinstance(t, str): return t107 if isinstance(t, str): return t
108 t = mx.DateTime.TimestampFromTicks(t)108 t = datetime.fromtimestap(t)
109 d = mx.DateTime.gmt() - t109 d = datetime.utcnow() - t
110110
111 # Aliasing the function doesn't work here with intltool...111 # Aliasing the function doesn't work here with intltool...
112 if d.days >= 365:112 if d.days >= 365:
113113
=== modified file 'tests/plugins/test/__init__.py'
--- tests/plugins/test/__init__.py 2012-02-10 10:06:59 +0000
+++ tests/plugins/test/__init__.py 2012-05-12 02:14:17 +0000
@@ -1,13 +1,11 @@
1import json
1from gwibber.microblog import util2from gwibber.microblog import util
2import json3from gwibber.time import parsetime
34
4import logging5import logging
5logger = logging.getLogger("Plugin Test")6logger = logging.getLogger("Plugin Test")
6logger.debug("Initializing.")7logger.debug("Initializing.")
78
8import re, mx.DateTime
9from gettext import lgettext as _
10
11PROTOCOL_INFO = {9PROTOCOL_INFO = {
12 "name": "Test",10 "name": "Test",
13 "version": 0.1,11 "version": 0.1,
@@ -40,7 +38,7 @@
40 m["service"] = "test"38 m["service"] = "test"
41 m["account"] = self.account["id"]39 m["account"] = self.account["id"]
42 if data.has_key("created_at"):40 if data.has_key("created_at"):
43 m["time"] = util.parsetime(data["created_at"])41 m["time"] = parsetime(data["created_at"])
44 m["source"] = data["source"]42 m["source"] = data["source"]
45 m["text"] = data["text"]43 m["text"] = data["text"]
4644
4745
=== added directory 'tests/python/unittests'
=== added file 'tests/python/unittests/__init__.py'
=== added file 'tests/python/unittests/test_time.py'
--- tests/python/unittests/test_time.py 1970-01-01 00:00:00 +0000
+++ tests/python/unittests/test_time.py 2012-05-12 02:14:17 +0000
@@ -0,0 +1,53 @@
1import unittest
2
3from gwibber.time import parsetime
4
5
6class TimeParseTest(unittest.TestCase):
7 def test_type(self):
8 # parsetime() should always return int seconds since the epoch.
9 self.assertTrue(isinstance(parsetime('2012-05-10T13:36:45'), int))
10
11 def test_parse_naive(self):
12 # ISO 8601 standard format without timezone.
13 self.assertEqual(parsetime('2012-05-10T13:36:45'), 1336682205)
14
15 def test_parse_utctz(self):
16 # ISO 8601 standard format with UTC timezone.
17 self.assertEqual(parsetime('2012-05-10T13:36:45 +0000'), 1336682205)
18
19 def test_parse_naive_altsep(self):
20 # ISO 8601 alternative format without timezone.
21 self.assertEqual(parsetime('2012-05-10 13:36:45'), 1336682205)
22
23 def test_parse_utctz_altsep(self):
24 # ISO 8601 alternative format with UTC timezone.
25 self.assertEqual(parsetime('2012-05-10T13:36:45 +0000'), 1336682205)
26
27 def test_bad_time_string(self):
28 # Odd unsupported format.
29 self.assertRaises(ValueError, parsetime, '2012/05/10 13:36:45')
30
31 def test_non_utc(self):
32 # Non-UTC timezones are get converted to UTC, before conversion to
33 # epoch seconds.
34 self.assertEqual(parsetime('2012-05-10T13:36:45 -0400'), 1336696605)
35
36 def test_nonstandard_twitter(self):
37 # Sigh. Twitter has to be different.
38 self.assertEqual(parsetime('Thu May 10 13:36:45 +0000 2012'),
39 1336682205)
40
41 def test_nonstandard_twitter_non_utc(self):
42 # Sigh. Twitter has to be different.
43 self.assertEqual(parsetime('Thu May 10 13:36:45 -0400 2012'),
44 1336696605)
45
46 def test_nonstandard_facebook(self):
47 # Sigh. Facebook gets close, but no cigar.
48 self.assertEqual(parsetime('2012-05-10T13:36:45+0000'), 1336682205)
49
50 def test_multiple_timezones(self):
51 # Multiple timezone strings are not supported.
52 self.assertRaises(ValueError, parsetime,
53 '2012-05-10T13:36:45 +0000 -0400')
054
=== modified file 'tests/python/utils/__init__.py'
--- tests/python/utils/__init__.py 2012-03-12 23:47:12 +0000
+++ tests/python/utils/__init__.py 2012-05-12 02:14:17 +0000
@@ -1,4 +1,7 @@
1import os, unittest, gettext, mx.DateTime1import time
2from datetime import datetime, timedelta
3
4import os, unittest
2from gi.repository import Gwibber5from gi.repository import Gwibber
3from hashlib import sha16from hashlib import sha1
47
@@ -7,34 +10,40 @@
7 os.environ["XDG_DATA_HOME"] = os.path.realpath (os.path.join (os.path.curdir, "..", "data"))10 os.environ["XDG_DATA_HOME"] = os.path.realpath (os.path.join (os.path.curdir, "..", "data"))
8 os.environ["XDG_CACHE_HOME"] = os.path.realpath (os.path.join (os.path.curdir, "..", "data"))11 os.environ["XDG_CACHE_HOME"] = os.path.realpath (os.path.join (os.path.curdir, "..", "data"))
9 self.utils = Gwibber.Utils ()12 self.utils = Gwibber.Utils ()
10 self.now = mx.DateTime.gmt()13 self.now = datetime.utcnow()
14
15 def _ago(self, **kws):
16 # Convert keywords into a timedelta, subtract that from utcnow, and
17 # return seconds since epoch.
18 timetup = (self.now - timedelta(**kws)).timetuple()
19 return time.mktime(timetup)
1120
12 def test_time_string_seconds (self):21 def test_time_string_seconds (self):
13 ts = self.utils.generate_time_string (self.now - 59 * mx.DateTime.oneSecond)22 ts = self.utils.generate_time_string(self._ago(seconds=59))
14 self.assertEqual (ts, 'a few seconds ago')23 self.assertEqual (ts, 'a few seconds ago')
1524
16 def test_time_string_minute (self):25 def test_time_string_minute (self):
17 ts = self.utils.generate_time_string (self.now - 60 * mx.DateTime.oneSecond)26 ts = self.utils.generate_time_string(self._ago(seconds=60))
18 self.assertEqual (ts, '1 minute ago')27 self.assertEqual (ts, '1 minute ago')
1928
20 def test_time_string_minutes (self):29 def test_time_string_minutes (self):
21 ts = self.utils.generate_time_string (self.now - 3559 * mx.DateTime.oneSecond)30 ts = self.utils.generate_time_string(self._ago(seconds=3559))
22 self.assertIn ('minutes ago', ts)31 self.assertIn ('minutes ago', ts)
2332
24 def test_time_string_hour (self):33 def test_time_string_hour (self):
25 ts = self.utils.generate_time_string (self.now - 3601 * mx.DateTime.oneSecond)34 ts = self.utils.generate_time_string(self._ago(seconds=3601))
26 self.assertEqual (ts, '1 hour ago')35 self.assertEqual (ts, '1 hour ago')
2736
28 def test_time_string_hours (self):37 def test_time_string_hours (self):
29 ts = self.utils.generate_time_string (self.now - 7201 * mx.DateTime.oneSecond)38 ts = self.utils.generate_time_string(self._ago(seconds=7201))
30 self.assertIn ('hours ago', ts)39 self.assertIn ('hours ago', ts)
3140
32 def test_time_string_day (self):41 def test_time_string_day (self):
33 ts = self.utils.generate_time_string (self.now - 86400 * mx.DateTime.oneSecond)42 ts = self.utils.generate_time_string(self._ago(seconds=86400))
34 self.assertEqual (ts, '1 day ago')43 self.assertEqual (ts, '1 day ago')
3544
36 def test_time_string_days (self):45 def test_time_string_days (self):
37 ts = self.utils.generate_time_string (self.now - 2 * mx.DateTime.oneDay)46 ts = self.utils.generate_time_string(self._ago(days=2))
38 self.assertIn ('days ago', ts)47 self.assertIn ('days ago', ts)
3948
40class AvatarPathTestCase (unittest.TestCase):49class AvatarPathTestCase (unittest.TestCase):