Zeitgeist Framework

Merge lp:~thekorn/zeitgeist/flexible.engine.backend into lp:zeitgeist/0.1

  1. flexible.engine.backend
  2. Merge into 0.8-python
Proposed by Markus Korn
Status: Merged
Approved by: Siegfried Gevatter
Approved revision: 1052
Merged at revision: not available
Proposed branch: lp:~thekorn/zeitgeist/flexible.engine.backend
Merge into: lp:zeitgeist/0.1
Diff against target: None lines
To merge this branch: bzr merge lp:~thekorn/zeitgeist/flexible.engine.backend
Reviewer Review Type Date Requested Status
Mikkel Kamstrup Erlandsen Approve
Siegfried Gevatter Approve
Seif Lotfy Pending
Zeitgeist Framework Team Pending
Review via email: mp+9499@code.launchpad.net
To post a comment you must log in.
Revision history for this message
Markus Korn (thekorn) wrote :

This branch implements the flexible engine framework for zeitgeist, as discussed on IRC recently.
The most important changes are:
  * get_default_engine() is now in _zeitgeist.engine instead of _zeitgeist.engine.engine
  * get_default_engine() either returns the running engine instance, or creates a new one. To get the type of the new engine it looks in the ZEITGEIST_ENGINE environment variable, if not set it uses a hardcoded default, which is storm for now
  * new engine implementations can be added by creating _zeitgeist/engine/SOMENAME_engine.py, where SOMENAME is the type of the engine (like sqlite3, sqlite3_querymancer, couchdb etc.)
  * all engine tests are running using the strom engine implementation

Revision history for this message
Siegfried Gevatter (rainct) wrote :

Just had a quick look at the diff, but looks good.

review: Approve
Revision history for this message
Mikkel Kamstrup Erlandsen (kamstrup) wrote :

Review: Approve
Looks good, but I didn't run it to double check

> --
> https://code.launchpad.net/~thekorn/zeitgeist/flexible.engine.backend/+merge/9499
> You are requested to review the proposed merge of lp:~thekorn/zeitgeist/flexible.engine.backend into lp:zeitgeist.
>

--
Cheers,
Mikkel

Revision history for this message
Mikkel Kamstrup Erlandsen (kamstrup) wrote :

Looks good, but I didn't do a test run of it

review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file '_zeitgeist/engine/__init__.py'
2--- _zeitgeist/engine/__init__.py 2008-11-19 18:43:27 +0000
3+++ _zeitgeist/engine/__init__.py 2009-07-31 09:33:22 +0000
4@@ -0,0 +1,49 @@
5+import os
6+import logging
7+
8+ENGINE_FALLBACK = "storm"
9+
10+_engine = None
11+def create_engine(engine_type=None):
12+ """ Creates an engine instance of the type defined by 'engine_type'.
13+ If 'engine_type' is None 'ENGINE_FALLBACK' is used.
14+ This function looks at _zeitgeist.engine to find the engine implementation.
15+ Each engine implementation has to follow the following conventions:
16+ 1.) it has to be in _zeitgeist/engine/SOMENAME_engine.py
17+ (where SOMENAME defines the type)
18+ 2.) the name of the class has to be ZeitgeistEngine and the class
19+ itself has to be a sublass of _zeitgeist.engine.engine_base.BaseEngine
20+ """
21+ global _engine
22+ if engine_type is None:
23+ engine_type = ENGINE_FALLBACK
24+ engine_type = engine_type.lower()
25+ if _engine is not None:
26+ running_type = engine.__module__.split(".").pop().lower()
27+ if not running_type == engine_type:
28+ raise RuntimeError(
29+ ("There is already a zeitgeist engine running. But this "
30+ "engine has another than the requested type "
31+ "(requested='%s', running='%s')" %(engine_type, running_type))
32+ )
33+ return _engine
34+ try:
35+ engine_cls = __import__(
36+ "_zeitgeist.engine.%s_engine" %engine_type,
37+ globals(), locals(), ["ZeitgeistEngine",], -1
38+ )
39+ except ImportError, err:
40+ logging.exception("Could not load engine implementation for %r" %engine_type)
41+ raise RuntimeError("Could not load engine implementation for %r" %engine_type)
42+ _engine = engine_cls.ZeitgeistEngine()
43+ return _engine
44+
45+def get_default_engine():
46+ """ Get the running engine instance or create a new one.
47+ To get the type of the new engine instance it looks at the 'ZEITGEIST_ENGINE'
48+ environment variable. If this is not defined, it uses the engine type
49+ defined by ENGINE_FALLBACK.
50+ """
51+ if _engine is not None:
52+ return _engine
53+ return create_engine(engine_type=os.environ.get("ZEITGEIST_ENGINE", ENGINE_FALLBACK))
54
55=== added file '_zeitgeist/engine/engine_base.py'
56--- _zeitgeist/engine/engine_base.py 1970-01-01 00:00:00 +0000
57+++ _zeitgeist/engine/engine_base.py 2009-07-31 06:35:29 +0000
58@@ -0,0 +1,144 @@
59+import time
60+import logging
61+import gobject
62+import sys
63+
64+from functools import wraps
65+
66+from _zeitgeist.lrucache import LRUCache
67+
68+logging.basicConfig(level=logging.DEBUG)
69+log = logging.getLogger("zeitgeist.engine.engine_base")
70+
71+def time_insert(function):
72+ @wraps(function)
73+ def wrapper(*args, **kwargs):
74+ t1 = time.time()
75+ result = function(*args, **kwargs)
76+ log.debug("Inserted %s items in %.5f s." % (len(result), time.time() - t1))
77+ return result
78+ return wrapper
79+
80+class BaseEngine(gobject.GObject):
81+
82+ ALLOWED_FILTER_KEYS = set(["name", "uri", "tags", "mimetypes",
83+ "source", "content", "application", "bookmarked"])
84+
85+ def __init__(self):
86+
87+ gobject.GObject.__init__(self)
88+
89+ self._apps = set()
90+ self._last_time_from_app = {}
91+ self._applications = LRUCache(100)
92+
93+ def insert_event(self, ritem, commit=True, force=False):
94+ """
95+ Inserts an item into the database. Returns a positive number on success,
96+ zero otherwise (for example, if the item already is in the
97+ database). In case the positive number is 1, the inserted event is new,
98+ in case it's 2 the event already existed and was updated (this only
99+ happens when `force' is True).
100+ """
101+
102+ # check for required items and make sure all items have the correct type
103+ EventDict.check_missing_items(ritem, True)
104+
105+ # FIXME: uri, content, source are now required items, the statement above
106+ # will raise a KeyError if they are not there. What about mimetype?
107+ # and why are we printing a warning and returning False here instead of raising
108+ # an error at all? - Markus Korn
109+ if not ritem["uri"].strip():
110+ raise ValueError("Discarding item without a URI: %s" % ritem)
111+ if not ritem["content"].strip():
112+ raise ValueError("Discarding item without a Content type: %s" % ritem)
113+ if not ritem["source"].strip():
114+ raise ValueError("Discarding item without a Source type: %s" % ritem)
115+ if not ritem["mimetype"].strip():
116+ raise ValueError("Discarding item without a mimetype: %s" % ritem)
117+ return 0
118+
119+ @time_insert
120+ def insert_events(self, items):
121+ """
122+ Inserts items into the database and returns those items which were
123+ successfully inserted. If an item fails, that's usually because it
124+ already was in the database.
125+ """
126+
127+ inserted_items = []
128+ for item in items:
129+ # This is always 0 or 1, no need to consider 2 as we don't
130+ # use the `force' option.
131+ if self.insert_event(item, commit=False):
132+ inserted_items.append(item)
133+ return inserted_items
134+
135+ def get_item(self, uri):
136+ """ Returns basic information about the indicated URI. As we are
137+ fetching an item, and not an event, `timestamp' is 0 and `use'
138+ and `app' are empty strings."""
139+
140+ raise NotImplementedError
141+
142+ def find_events(self, min=0, max=sys.maxint, limit=0,
143+ sorting_asc=True, mode="event", filters=(), return_mode=0):
144+ """
145+ Returns all items from the database between the indicated
146+ timestamps `min' and `max'. Optionally the argument `tags'
147+ may be used to filter on tags or `mimetypes' to filter on
148+ mimetypes.
149+
150+ Parameter `mode' can be one of "event", "item" or "mostused".
151+ The first mode returns all events, the second one only returns
152+ the last event when items are repeated and the "mostused" mode
153+ is like "item" but returns the results sorted by the number of
154+ events.
155+
156+ Parameter `filters' is an array of structs containing: (text
157+ to search in the name, text to search in the URI, tags,
158+ mimetypes, source, content). The filter between characteristics
159+ inside the same struct is of type AND (all need to match), but
160+ between diferent structs it is OR-like (only the conditions
161+ described in one of the structs need to match for the item to
162+ be returned).
163+
164+ Possible values for return_mode, which is an internal variable
165+ not exposed in the API:
166+ - 0: Return the events/items.
167+ - 1: Return the amount of events/items which would be returned.
168+ - 2: Return only the applications for the matching events.
169+ """
170+ raise NotImplementedError
171+
172+ def update_items(self, items):
173+ raise NotImplementedError
174+
175+ def delete_items(self, items):
176+ raise NotImplementedError
177+
178+ def get_types(self):
179+ """
180+ Returns a list of all different types in the database.
181+ """
182+ raise NotImplementedError
183+
184+ def get_tags(self, min_timestamp=0, max_timestamp=0, limit=0, name_filter=""):
185+ """
186+ Returns a list containing tuples with the name and the number of
187+ occurencies of the tags matching `name_filter', or all existing
188+ tags in case it's empty, sorted from most used to least used. `limit'
189+ can base used to limit the amount of results.
190+
191+ Use `min_timestamp' and `max_timestamp' to limit the time frames you
192+ want to consider.
193+ """
194+ raise NotImplementedError
195+
196+ def get_last_insertion_date(self, application):
197+ """
198+ Returns the timestamp of the last item which was inserted
199+ related to the given application. If there is no such record,
200+ 0 is returned.
201+ """
202+ raise NotImplementedError
203
204=== modified file '_zeitgeist/engine/remote.py'
205--- _zeitgeist/engine/remote.py 2009-07-20 17:10:41 +0000
206+++ _zeitgeist/engine/remote.py 2009-07-23 17:37:06 +0000
207@@ -21,7 +21,7 @@
208 import dbus.service
209 import logging
210
211-from _zeitgeist.engine.engine import get_default_engine
212+from _zeitgeist.engine import get_default_engine
213 from zeitgeist.dbusutils import DBusInterface
214 from _zeitgeist.singleton import SingletonApplication
215
216
217=== renamed file '_zeitgeist/engine/base.py' => '_zeitgeist/engine/storm_base.py'
218=== renamed file '_zeitgeist/engine/engine.py' => '_zeitgeist/engine/storm_engine.py'
219--- _zeitgeist/engine/engine.py 2009-07-29 12:16:43 +0000
220+++ _zeitgeist/engine/storm_engine.py 2009-07-31 06:35:29 +0000
221@@ -25,7 +25,6 @@
222 import sys
223 import os
224 import gettext
225-import gobject
226 import logging
227 from xdg import BaseDirectory
228 from xdg.DesktopEntry import DesktopEntry
229@@ -34,34 +33,22 @@
230 except ImportError:
231 import sqlite3
232
233-from _zeitgeist.engine.base import *
234-from _zeitgeist.lrucache import LRUCache
235+from _zeitgeist.engine.storm_base import *
236+from _zeitgeist.engine.engine_base import BaseEngine
237 from zeitgeist.dbusutils import EventDict
238
239 logging.basicConfig(level=logging.DEBUG)
240 log = logging.getLogger("zeitgeist.engine")
241
242-class ZeitgeistEngine(gobject.GObject):
243-
244- ALLOWED_FILTER_KEYS = set(["name", "uri", "tags", "mimetypes",
245- "source", "content", "application", "bookmarked"])
246-
247- def __init__(self, storm_store):
248-
249- gobject.GObject.__init__(self)
250-
251- assert storm_store is not None
252- self.store = storm_store
253- self._apps = set()
254- self._last_time_from_app = {}
255- self._applications = LRUCache(100)
256-
257- '''
258- path = BaseDirectory.save_data_path("zeitgeist")
259- database = os.path.join(path, "zeitgeist.sqlite")
260- self.connection = self._get_database(database)
261- self.cursor = self.connection.cursor()
262- '''
263+class ZeitgeistEngine(BaseEngine):
264+
265+ def __init__(self, store=None):
266+ super(ZeitgeistEngine, self).__init__()
267+ if store is not None:
268+ self.store = store
269+ else:
270+ self.store = get_default_store()
271+ assert self.store is not None
272
273 def _get_ids(self, uri, content, source):
274 uri_id = URI.lookup_or_create(uri).id if uri else None
275@@ -203,21 +190,9 @@
276 successfully inserted. If an item fails, that's usually because it
277 already was in the database.
278 """
279-
280- inserted_items = []
281-
282- time1 = time.time()
283- for item in items:
284- # This is always 0 or 1, no need to consider 2 as we don't
285- # use the `force' option.
286- if self.insert_event(item, commit=False):
287- inserted_items.append(item)
288- self.store.commit()
289- time2 = time.time()
290- log.debug("Inserted %s items in %.5f s." % (len(inserted_items),
291- time2 - time1))
292-
293- return inserted_items
294+ result = super(ZeitgeistEngine, self).insert_events(items)
295+ self.store.commit()
296+ return result
297
298 def get_item(self, uri):
299 """ Returns basic information about the indicated URI. As we are
300@@ -499,10 +474,3 @@
301 ORDER BY start DESC LIMIT 1
302 """, (application,)).get_one()
303 return query[0] if query else 0
304-
305-_engine = None
306-def get_default_engine():
307- global _engine
308- if not _engine:
309- _engine = ZeitgeistEngine(get_default_store())
310- return _engine
311
312=== modified file 'test/benchmarks.py'
313--- test/benchmarks.py 2009-07-13 07:43:29 +0000
314+++ test/benchmarks.py 2009-07-23 16:58:55 +0000
315@@ -5,10 +5,10 @@
316 import os
317 sys.path.insert(0, os.path.join(os.path.dirname(__file__), ".."))
318
319-from _zeitgeist.engine.base import create_store, set_store
320-from _zeitgeist.engine import base
321+from _zeitgeist.engine.storm_base import create_store, set_store
322+from _zeitgeist.engine import storm_base as base
323 from zeitgeist.datamodel import *
324-from _zeitgeist.engine.engine import ZeitgeistEngine
325+from _zeitgeist.engine.storm_engine import ZeitgeistEngine
326
327 from time import time
328 import unittest
329
330=== modified file 'test/engine-engine-test.py'
331--- test/engine-engine-test.py 2009-07-15 21:36:41 +0000
332+++ test/engine-engine-test.py 2009-07-23 16:58:55 +0000
333@@ -5,10 +5,10 @@
334 import os
335 sys.path.insert(0, os.path.join(os.path.dirname(__file__), ".."))
336
337-from _zeitgeist.engine.base import create_store, set_store
338-from _zeitgeist.engine import base
339+from _zeitgeist.engine.storm_base import create_store, set_store
340+from _zeitgeist.engine import storm_base as base
341 from zeitgeist.datamodel import *
342-from _zeitgeist.engine.engine import ZeitgeistEngine
343+from _zeitgeist.engine.storm_engine import ZeitgeistEngine
344
345 import unittest
346 import tempfile
347
348=== modified file 'zeitgeist-daemon'
349--- zeitgeist-daemon 2009-07-20 17:10:41 +0000
350+++ zeitgeist-daemon 2009-07-23 17:37:06 +0000
351@@ -33,11 +33,8 @@
352 gettext.install("zeitgeist", _config.localedir, unicode=1)
353 logging.basicConfig(level=logging.DEBUG)
354
355-from _zeitgeist.engine.engine import get_default_engine
356 from _zeitgeist.engine.remote import RemoteInterface
357
358-engine = get_default_engine()
359-
360 dbus.mainloop.glib.DBusGMainLoop(set_as_default=True)
361 mainloop = gobject.MainLoop()
362

Subscribers

People subscribed via source and target branches