Merge lp:~robru/gwibber/discovery into lp:~barry/gwibber/py3

Proposed by Robert Bruce Park
Status: Merged
Merged at revision: 1416
Proposed branch: lp:~robru/gwibber/discovery
Merge into: lp:~barry/gwibber/py3
Diff against target: 154 lines (+35/-59)
4 files modified
gwibber/gwibber/protocols/flickr.py (+1/-1)
gwibber/gwibber/tests/test_protocols.py (+11/-15)
gwibber/gwibber/utils/modules.py (+0/-43)
gwibber/gwibber/utils/protocol.py (+23/-0)
To merge this branch: bzr merge lp:~robru/gwibber/discovery
Reviewer Review Type Date Requested Status
Barry Warsaw Pending
Review via email: mp+122995@code.launchpad.net

Description of the change

Ugh, today was a disaster :-/

So what I've done here is merged find_all_protocols and Base protocol class into gwibber.utils.protocol. I've also done the bare minimum to convert find_all_protocols into ProtocolManager.find_all. I ran into a number of seemingly intractable problems:

1. Despite Base class no longer living in protocols directory, inspect.getmembers is still managing to yield it from find_all. My guess would be that this is happening because the flickr plugin imports Base into it's own namespace, so inspect finds it in that namespace and then yields it. So the 'value is not Base' test is still necessary, defeating much of what I set out to do today. The only way I can see around this is to give up on the idea of importing all classes, and just import the class that matches the filename (or just live with it as-is, that's probably not so bad either).

2. My idea that the ProtocolManager would actually instantiate all plugins in it's __init__ method is another utter failure, because the plugins need accounts in order to be instantiated, and the PluginManager doesn't have any access to the accounts in order to actually do that. So the best we're going to get from this is a dict mapping basenames to classes (not instances), which I am about to implement. It's going to break your elegant generator, but unfortunately that is necessary because the other bits of gwibber are going to need to do 'ProtocolManager["flickr"]' and not 'for p in ProtocolManager.find_all(): if str(p) != "flickr": continue' or some other heinous mess.

Anyway, please review & merge this, and then we can discuss how to proceed with regards to the dispatcher interacting with plugins. I'm thinking that the Account class should contain a reference to the instance of the protocol that it belongs to, but in particular we need to figure out what the lifetime and scope of protocol instances is going be. Eg, can we design it so that the instance that represents each acount is persistent in memory (similar to my @memoize magic), or is it ok to have multiple different instances of the protocol class representing a given account (eg, if two different threads happen to instantiate their own Flickr plugin for the same account, is that ok, or is that going to conflict with each other). I guess what I'm really trying to say is that we need to audit this code for thread safety.

Thanks!

To post a comment you must log in.
lp:~robru/gwibber/discovery updated
1419. By Robert Bruce Park

Make ProtocolManager a dict mapping protocol names to their classes.

Revision history for this message
Robert Bruce Park (robru) wrote :

Barry, I can't figure out why the tests aren't passing here, it works successfully when I access it from an interactive python session. Something about the unittest environment is breaking something else that's going on, but I can't figure out what because I haven't changed much from your code that did work, the biggest change is that it's executing the code immediately instead of as a generator.

So sorry for mp'ing broken code, hopefully there's something obvious that I've overlooked because I'm exhausted. Please have a look!

lp:~robru/gwibber/discovery updated
1420. By Robert Bruce Park

Drop @singleton decorator to make tests pass.

Revision history for this message
Barry Warsaw (barry) wrote :
Download full text (9.3 KiB)

On Sep 06, 2012, at 03:22 AM, Robert Bruce Park wrote:

>Ugh, today was a disaster :-/

Bummer! Today will surely be better.

>1. Despite Base class no longer living in protocols directory,
>inspect.getmembers is still managing to yield it from find_all. My guess
>would be that this is happening because the flickr plugin imports Base into
>it's own namespace, so inspect finds it in that namespace and then yields
>it.

Yep, that's exactly what's happening.

>So the 'value is not Base' test is still necessary, defeating much of what I
>set out to do today. The only way I can see around this is to give up on the
>idea of importing all classes, and just import the class that matches the
>filename (or just live with it as-is, that's probably not so bad either).

Actually, I realize that instead of iterating over all members of the module,
you should just iterate over the names in the module's __all__. That's the
intent of __all__ anyway, to explicitly export the public members of a module.
Of course it means that __all__ has to be kept up-to-date, but I think that's
a good idea anyway.

The loop will still have to do the subclass check, but it won't need to check
for non-Base-y-ness.

>2. My idea that the ProtocolManager would actually instantiate all plugins in
>it's __init__ method is another utter failure, because the plugins need
>accounts in order to be instantiated, and the PluginManager doesn't have any
>access to the accounts in order to actually do that.

We discussed this a bit in IRC. I definitely need to read up on libaccounts,
but I think we've decided that it's possible to have multiple instances of the
protocol classes with different account information. E.g. you might have two
Twitter accounts. So yeah, ProtocolManager can only find and record the
protocol classes, though depending on the outcome of future designs, it could
possibly provide an API for more convenient instantiation of the classes. I
think we're agreed that there need be only one global "account manager" per
gwibber-service.

We'll cross that bridge later though.

>So the best we're going to get from this is a dict mapping basenames to
>classes (not instances), which I am about to implement. It's going to break
>your elegant generator, but unfortunately that is necessary because the other
>bits of gwibber are going to need to do 'ProtocolManager["flickr"]' and not
>'for p in ProtocolManager.find_all(): if str(p) != "flickr": continue' or
>some other heinous mess.

I'm also not psyched about having ProtocolManager be a subclass of dict. In
general, I don't like code that does that just to provide a getitem API to
clients, but the class itself isn't intended to be used everywhere a dict is
to be used. It seems like an inappropriate use of inheritance. We use it in
the Account class (which I still don't like, but oh well), but I don't think
we should use that for ProtocolManager.

Or to put it in OOP parlance: a ProtocolManager *has a* dict, but not *is a*
dict. It's generally frowned upon to use inheritance for *has a*
relationships. I'd like to rewrite that for the merge.

One other thing I've changed. The protocol classes themselves have a 'name'
attribut...

Read more...

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== modified file 'gwibber/gwibber/protocols/flickr.py'
--- gwibber/gwibber/protocols/flickr.py 2012-09-05 22:09:44 +0000
+++ gwibber/gwibber/protocols/flickr.py 2012-09-06 15:16:41 +0000
@@ -23,11 +23,11 @@
23import logging23import logging
2424
25from gwibber.errors import AuthorizationError25from gwibber.errors import AuthorizationError
26from gwibber.protocols.base import Base
27from gwibber.utils.authentication import Authentication26from gwibber.utils.authentication import Authentication
28from gwibber.utils.download import get_json27from gwibber.utils.download import get_json
29from gwibber.utils.results import Results28from gwibber.utils.results import Results
30from gwibber.utils.time import parsetime29from gwibber.utils.time import parsetime
30from gwibber.utils.protocol import Base
3131
3232
33log = logging.getLogger('gwibber.service')33log = logging.getLogger('gwibber.service')
3434
=== modified file 'gwibber/gwibber/tests/test_protocols.py'
--- gwibber/gwibber/tests/test_protocols.py 2012-09-05 22:09:44 +0000
+++ gwibber/gwibber/tests/test_protocols.py 2012-09-06 15:16:41 +0000
@@ -25,12 +25,12 @@
25import unittest25import unittest
26import threading26import threading
2727
28from gwibber.errors import AuthorizationError28from gwibber.testing.mocks import FakeData
29from gwibber.protocols.base import Base
30from gwibber.protocols.flickr import Flickr29from gwibber.protocols.flickr import Flickr
31from gwibber.testing.mocks import FakeData
32from gwibber.utils.logging import initialize30from gwibber.utils.logging import initialize
33from gwibber.utils.modules import find_all_protocols31from gwibber.errors import AuthorizationError
32from gwibber.utils.protocol import Base, ProtocolManager
33
3434
35try:35try:
36 # Python 3.336 # Python 3.3
@@ -43,20 +43,16 @@
43 """Test the protocol finder."""43 """Test the protocol finder."""
4444
45 def test_find_all_protocols(self):45 def test_find_all_protocols(self):
46 # Search for and return all of the discovered classes implementing the46 # ProtocolManager knows them all.
47 # known protocols.47 manager = ProtocolManager()
48 flickr_class = None48 self.assertTrue('flickr' in manager)
49 for protocol_class in find_all_protocols():49 self.assertEqual(manager['flickr'], Flickr)
50 if protocol_class.name == 'Flickr':
51 flickr_class = protocol_class
52 break
53 else:
54 raise AssertionError('Flickr protocol class not found')
55 self.assertEqual(flickr_class, Flickr)
5650
57 def test_doesnt_find_base(self):51 def test_doesnt_find_base(self):
58 # Make sure that the base protocol class isn't returned.52 # Make sure that the base protocol class isn't returned.
59 for protocol_class in find_all_protocols():53 manager = ProtocolManager()
54 self.assertFalse('base' in manager)
55 for protocol_class in manager.values():
60 self.assertNotEqual(protocol_class, Base)56 self.assertNotEqual(protocol_class, Base)
61 self.assertIsNotNone(protocol_class.name)57 self.assertIsNotNone(protocol_class.name)
6258
6359
=== removed file 'gwibber/gwibber/utils/modules.py'
--- gwibber/gwibber/utils/modules.py 2012-09-05 19:22:46 +0000
+++ gwibber/gwibber/utils/modules.py 1970-01-01 00:00:00 +0000
@@ -1,43 +0,0 @@
1# Copyright (C) 2012 Canonical Ltd
2#
3# This program is free software: you can redistribute it and/or modify
4# it under the terms of the GNU General Public License version 2 as
5# published by the Free Software Foundation.
6#
7# This program is distributed in the hope that it will be useful,
8# but WITHOUT ANY WARRANTY; without even the implied warranty of
9# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
10# GNU General Public License for more details.
11#
12# You should have received a copy of the GNU General Public License
13# along with this program. If not, see <http://www.gnu.org/licenses/>.
14
15"""Utilities for finding things in modules."""
16
17__all__ = [
18 'find_all_protocols',
19 ]
20
21
22import os
23import inspect
24import importlib
25
26from pkg_resources import resource_listdir
27
28from gwibber.protocols.base import Base
29
30
31def find_all_protocols():
32 for filename in resource_listdir('gwibber', 'protocols'):
33 basename, extension = os.path.splitext(filename)
34 if extension != '.py':
35 continue
36 module_path = 'gwibber.protocols.' + basename
37 module = importlib.import_module(module_path)
38 # Now, look at all the names in the just imported module. Return all
39 # class objects that inherit from the base protocol class. Be sure
40 # not to return the base protocol class itself!
41 for name, value in inspect.getmembers(module, inspect.isclass):
42 if issubclass(value, Base) and value is not Base:
43 yield value
440
=== renamed file 'gwibber/gwibber/protocols/base.py' => 'gwibber/gwibber/utils/protocol.py'
--- gwibber/gwibber/protocols/base.py 2012-09-05 14:18:12 +0000
+++ gwibber/gwibber/utils/protocol.py 2012-09-06 15:16:41 +0000
@@ -16,9 +16,16 @@
1616
17__all__ = [17__all__ = [
18 'Base',18 'Base',
19 'ProtocolManager',
19 ]20 ]
2021
2122
23import os
24import inspect
25import importlib
26
27from pkg_resources import resource_listdir
28
22from threading import Thread29from threading import Thread
2330
2431
@@ -89,3 +96,19 @@
89 def _locked_login(self, old_token):96 def _locked_login(self, old_token):
90 """Login operation stub. This must be implemented by subclasses."""97 """Login operation stub. This must be implemented by subclasses."""
91 raise NotImplementedError98 raise NotImplementedError
99
100
101class ProtocolManager(dict):
102 """Discover and instantiate all protocols."""
103 def __init__(self):
104 for filename in resource_listdir('gwibber', 'protocols'):
105 basename, extension = os.path.splitext(filename)
106 if extension != '.py':
107 continue
108 module_path = 'gwibber.protocols.' + basename
109 module = importlib.import_module(module_path)
110 # Now, look at all the names in the just imported module. Return all
111 # class objects that inherit from the Base protocol class.
112 self.update((name.lower(), obj) for (name, obj)
113 in inspect.getmembers(module, inspect.isclass)
114 if issubclass(obj, Base) and obj is not Base)

Subscribers

People subscribed via source and target branches