Merge lp:~dylanmccall/harvest/harvest-dylan-m into lp:harvest

Proposed by Dylan McCall
Status: Merged
Merged at revision: 186
Proposed branch: lp:~dylanmccall/harvest/harvest-dylan-m
Merge into: lp:harvest
Diff against target: 806 lines (+706/-1)
11 files modified
INSTALL (+1/-1)
harvest/common/url_tools.py (+32/-0)
harvest/filters/__init__.py (+23/-0)
harvest/filters/containers.py (+97/-0)
harvest/filters/filters.py (+318/-0)
harvest/opportunities/filters.py (+53/-0)
harvest/opportunities/urls.py (+4/-0)
harvest/opportunities/views.py (+34/-0)
harvest/opportunities/wrappers.py (+90/-0)
harvest/settings.py (+4/-0)
harvest/templates/opportunities/opportunities_filter.html (+50/-0)
To merge this branch: bzr merge lp:~dylanmccall/harvest/harvest-dylan-m
Reviewer Review Type Date Requested Status
Daniel Holbach Approve
James Westby Approve
Review via email: mp+28262@code.launchpad.net

Description of the change

Introducing a new view at /opportunities/filter, which aims to replace all existing views in the future.

This view presents all packages and all opportunities tracked by Harvest, where the user can filter them using a list of properties. First the list of source packages is filtered, then the list of opportunities related to those packages is filtered.

User interface is just a proof of concept so far, with very few avenues for interaction (beyond toggling some filters in a rudimentary way). The back-end lets us quickly define new properties to filter packages and opportunities by, and these are instantly reflected in the output.

Everything happens through static pages at the moment, with parameters passed in the query string. Any interaction that involves clicking a link, including expanding a package to show its related opportunities, is a full page load away.

To post a comment you must log in.
Revision history for this message
Dylan McCall (dylanmccall) wrote :

Oh, I forgot. This also adds a dependency on debug-toolbar. That change can be reverted (it's just in settings.py and INSTALL), but it also doesn't hurt. It's very useful for tracking how changes affect performance.

Note that the middleware adds a whole ton of data to the page as it is sent (when enabled), so it does have a performance hit of its own. It only turns on if it's being accessed through 127.0.0.1.

Revision history for this message
Daniel Holbach (dholbach) wrote :

> Oh, I forgot. This also adds a dependency on debug-toolbar. That change can be
> reverted (it's just in settings.py and INSTALL), but it also doesn't hurt.
> It's very useful for tracking how changes affect performance.
>
> Note that the middleware adds a whole ton of data to the page as it is sent
> (when enabled), so it does have a performance hit of its own. It only turns on
> if it's being accessed through 127.0.0.1.

Maybe we can ask people to add that stuff to local_settings.py or we require YES_I_AM_HARVEST_HACKER to be True? We could also test if debug_toolbar can be imported.

About the rest of the new code: HOLY COW! You put quite a bit of work into this! :)

FilterContainer:
~~~~~~~~~~~~~~~~
minor only, but the description of set_filters() says "Add a set of filters to be children of this one." although self.filters_dict is reset at the beginning. Would an add_*() function be useful or should the description just be modified a bit?

I think I'd rename get() to find(), not sure which is more common. Both work for me. :-)

113 + #the first bit is the one this instance stores
114 + if name_bits[0] in self.filters_dict:
115 + result = self.filters_dict[name_bits[0]]
116 + else:
117 + result = None

Lines 116 and 117 can be removed.

119 + if isinstance(result, FilterContainer) and len(name_bits)>1:
120 + return result.get(name_bits[1]) #send remaining bits to child
121 + else:
122 + #we have reached the end of the list. Return the result (which may be None)
123 + return result

121 can be removed and 123 unidented one level.

FilterSystem:
~~~~~~~~~~~~~

Does every call to get_parameters() produce a copy? Do we need a get()ter there?

Filter:
~~~~~~~

Do we need explicit getters and setters() for the very simple members of Filter?

Do you think it's useful to add a few very quick examples to some of descriptions? ie: to me it wasn't immediately obvious what the "toggling" is about.

Revision history for this message
Daniel Holbach (dholbach) wrote :

(minor) Maybe FilterSystem.update_http() could be called FilterSystem.update_from_http() to make clear what is updated based on which data?

Can FilterSystem and FilterContainer be merged? Are they used separately and in different ways? I must admit FilterSystem is not quite clear to me yet. Still digging through it. :)

Revision history for this message
Daniel Holbach (dholbach) wrote :

Filter: I'm not sure what get_value() and get_value_string*() and get_parameters_for_value() are for and where the differences lie.

Revision history for this message
Daniel Holbach (dholbach) wrote :

Can we get PackageListWrapper.*visible_packages* and PackageListWrapper.*hidden_packages* into opportunity/views.py:opportunities_filter()? We could just pass hidden_packages and visible_packages instead of packages_list. Similarly we could do something for PackageWrapper. Also could we probably do some of the expand logic in javascript. What do you think?

Revision history for this message
James Westby (james-w) wrote :

Wow!

Thanks Dylan.

31 + url_params = list()

seems to be superfluous.

Would there ever be a desire for

36 +def current_url_with_parameters(request, new_params_dict):

to remove parameters? That can be something that is added later if needed
though.

46 \ No newline at end of file
453 \ No newline at end of file
673 \ No newline at end of file

Fixing those would be nice.

149 + #note that this currently stores parameters that aren't relevant to the FilterSystem
150 + #we need to do that so we don't eat them in get_url_with_parameters
151 + self.set_parameters(request.GET)

is that still needed with the code to get the original params in get_url_with_parameters?

Where you mark a method as abstract and don't implement in then consider

    raise NotImplementedError(self.<abstract_method>)

which will give an error if the subclass doesn't implement it, instead of
silently doing nothing.

244 + @param choices_dict: Optional value to be used instead of internal value

that's not a parameter to that method, same in the next method.

Where you use mark_safe you should add a comment stating why that string is
known to be safe.

Do the default parameters make sense? I'm not sure we should be defaulting to
"ged" at least.

Overall this is some seriously great code, nice work!

Thanks,

James

review: Approve
Revision history for this message
Dylan McCall (dylanmccall) wrote :
Download full text (8.2 KiB)

Yay! Thanks for the comments. I'll respond to them here.

There is a bit of superfluous initialization (by Python standards) going
on, indeed. Thanks for pointing them out! I was still in Vala mode when
I started this, desperately clinging to the comfort of strong typing ;)

So, in order from the top!

---FilterContainer---

        FilterContainer is currently used by two classes: FilterGroup
        under filters/filters.py, and FilterSystem under
        filters/containers.py. It is a bit wacky at the moment.
        FilterSystem is the one single root container that all the
        filters go inside of. It should be The object that the rest of
        the application uses to manipulate filters. The current HTTP
        request is handed to that object and it sorts out the rest.
        Thankfully, that means the thing can be poked at repeatedly
        until the wackiness subsides, without affecting the rest of the
        code.
        FilterGroup is used to group filters that work with a specific
        collection of objects (there's a pkg and opp group at the
        moment). It is possible to enable and disable filters inside a
        FilterGroup.

        set_filters() probably made sense the way it was at some point
        in the past. It will still only ever be called once, but I
        needn't enforce that; it just adds complexity for no particular
        reason. I renamed set_filters to add_filters and I'm
        initializing self.filters_dict under __init__. All these
        functions starting with set_ and get_ were making my head hurt,
        anyway.

        get_parameters(), as it turns out, was just redundant and isn't
        used anywhere any more. I'll strip that out. Similar
        functionality happens behind the scenes in url_tools.py, with
        current_url_with_parameters(). Come to think of it, there's _a
        lot_ of copying going on for one request. It can probably be
        sped up somewhere.

        FilterContainer.get() does imply a similarity to the get()
        method of many other types, even though it is definitely not
        like those. Good point. Changed it to find()!

        Daniel, you're awesome at naming functions! :)

        Thanks for spotting the leftover set_parameters stuff, James.
        It's all gone now! (*phew*)

---Filters---

        The default parameters are just for testing purposes, and indeed
        make no sense. I'm using gedit as a consistent query to test
        against, so when it looks exactly like the mockup I'll know! (It
        also needs some kind of weird default, because it doesn't handle
        big lists of packages very gracefully).
        pkg:name is set as such because there is no interface to change
        it yet, except editing the URL. Normally, of course, it would be
        blank.
        (Unfortunately, said interface won't be pretty. Solving it with
        Javascript is easy, but doing it without needs a pretty
        convoluted HTML form. I may just do the Javascript solution for
        now).

        get_value(), get_value_s...

Read more...

201. By Dylan McCall

Cleanup, as discussed at <https://code.edge.launchpad.net/~dylanmccall/harvest/harvest-dylan-m/+merge/28262>

Added newlines at end of files, where they were missing.

harvest/filters:
Finished docstrings
Removing "#abstract" comment where it makes no sense

harvest/filters/containers.py:
Renamed some methods for clarity

harvest/filters/filters.py:
raising exception for methods that need to be implemented
Rejigged get_values, get_value_string and get_parameters_for_value. Now a filter can be serialized and it's all a little bit simpler

202. By Dylan McCall

Fixed a bug where serialize_value, given an empty set for value, used the current value instead.

Revision history for this message
Daniel Holbach (dholbach) wrote :
Download full text (3.2 KiB)

Thanks a lot for your continued work on this and sorry for not replying earlier. I've been quite busy with other things. Sorry.

> The setters and getters found themselves in such quantity
> because the implementation of these things can fluctuate. A lot
> of that fluctuation has been reduced with the version you're
> seeing here, though. The other reason is I find it elegant to
> have the outside world only access an object through methods;
> never through properties. (That way the rules for accessing a
> given property are self-documenting). If that's silly, let me
> know!

In cases where setters and getters just do the minimal amount of work ("self.bla = bla" and "return bla"), I'd expect that accessing the object's property would just work. If additional work needs to be done, I'd try to do stuff in the constructor or in a function that does some kind of computation or other work.

I don't know if that's applicable here.

> The reason I have visible_packages and hidden_packages in
> wrappers.py is that there will be a bunch of other stuff there
> in the near future. For example, stuff that summarizes the two
> collections of packages. Some of this demands processing that
> may or may not happen, so it makes sense to call the appropriate
> functions from the template as appropriate (I think…).

Can you maybe explain what kind of summary you'd like to see there?

I personally feel it's a bit more work to get it right, but it might be beneficial to tune the queries in views.py right, which would also save us from loading too many objects into memory.

> PackageWrapper is similar; it has little at the moment, but in
> the future could be used to access what categories of
> opportunities lie within, including hidden ones. Throwing that
> logic in the template (even with template tags) feels like a
> horrible act, and probably wouldn't be as efficient.
> Having said that, PackageWrapper feels a bit more wrong because
> there is already a perfectly good SourcePackage object we can
> add data to. I just feel squeamish throwing that stuff directly
> at a SourcePackage model instance. I know it won't save it to
> the database or anything, but it feels wrong somehow.
>
> I kept an eye on performance when I put that together. It does
> grab the entire list of source packages from the database and
> turn them into new Python objects, but in the end we're just
> doing the same database hit that would happen later. Debug
> Toolbar says we hit the database once for each model; nothing
> blatantly redundant is happening. Always room for better
> performance, though!

I was under the impression that we wouldn't have to have all opportunities and source packages in memory if the query just asked for a specific subset. Maybe I'm wrong. I'll go and find out.

> Okay, that's a lot of writing, but this has been really helpful to get
> my thoughts straightened. It feels a lot smoother than it did this
> morning!

Y...

Read more...

Revision history for this message
Dylan McCall (dylanmccall) wrote :

I did some pondering and poking, and I'm further convinced to change my use of accessors. Turns out the most Pythonic way is to use plain instance variables and implement property() as appropriate, which lets us specify our own getters and setters for those variables (or just a getter). That Java course nearly corrupted me!

There is a bit of divergence in filters.py in the gsoc-client-stuff branch, so I'm a little reluctant to make that change in this branch. (Merge conflicts — even simple ones — invariably give me headaches). Are you okay if I do it in a new branch from gsoc-client-stuff?

> I was under the impression that we wouldn't have to have all
> opportunities and source packages in memory if the query just asked
> for a specific subset. Maybe I'm wrong. I'll go and find out.

I mean to say that all the source packages which have been met by the package filtergroup are examined in a similar way. It isn't an extensive thing, but we end up accessing the queryset, which gives us a list of all the references it has found, both hidden and visible packages. (The opportunities remain a queryset until the template asks for them). I should move that to be smarter about hidden packages, only storing their count via a query.

With gsoc-client-stuff, I'm working on asynchronously loading results, package info and hidden packages. This means implementing new views on the same query data as before. So, that's why I am a little picky about having one object to manage all that; all these views can just talk to the same thing in a somewhat balanced way. Having said that, it's mostly a stop-gap and if all goes well it should be really easy to yank it out for something quicker and more cool once the dust is settled. To be honest I'm not sure what the final result is going to look like in views.py, so I am reluctant to devote too much to a specific approach yet ;)

Revision history for this message
Daniel Holbach (dholbach) wrote :

> That Java course nearly
> corrupted me!

BIG HUGS! I'm sure you'll survive, though! :-)

> There is a bit of divergence in filters.py in the gsoc-client-stuff branch, so
> I'm a little reluctant to make that change in this branch. (Merge conflicts —
> even simple ones — invariably give me headaches). Are you okay if I do it in a
> new branch from gsoc-client-stuff?

Sure. I'm also happy for small branches and small merge proposals to go in as it should be much quicker, easier to review, etc.

> > I was under the impression that we wouldn't have to have all
> > opportunities and source packages in memory if the query just asked
> > for a specific subset. Maybe I'm wrong. I'll go and find out.
>
> I mean to say that all the source packages which have been met by the package
> filtergroup are examined in a similar way. It isn't an extensive thing, but we
> end up accessing the queryset, which gives us a list of all the references it
> has found, both hidden and visible packages. (The opportunities remain a
> queryset until the template asks for them). I should move that to be smarter
> about hidden packages, only storing their count via a query.

As I told you in a separate discussion my main concern was merely the complexity of new classes we add. If we (and particularly new contributors) try to fix upcoming bugs, it'll be harder and harder to dive through various classes which sometimes have very similar names to figure out where something is done.

Having said that, we can surely merge this for now and simplify over time and as we get a better idea of what we want to get done.

> With gsoc-client-stuff, I'm working on asynchronously loading results, package
> info and hidden packages. This means implementing new views on the same query
> data as before. So, that's why I am a little picky about having one object to
> manage all that; all these views can just talk to the same thing in a somewhat
> balanced way. Having said that, it's mostly a stop-gap and if all goes well it
> should be really easy to yank it out for something quicker and more cool once
> the dust is settled. To be honest I'm not sure what the final result is going
> to look like in views.py, so I am reluctant to devote too much to a specific
> approach yet ;)

Gotcha. Thanks for your work on this and consideration.

review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== modified file 'INSTALL'
--- INSTALL 2010-03-02 10:42:19 +0000
+++ INSTALL 2010-06-27 21:07:25 +0000
@@ -1,4 +1,4 @@
11. sudo apt-get install python-django python-launchpadlib python-django-openid-auth bzr11. sudo apt-get install python-django python-launchpadlib python-django-openid-auth bzr python-django-debug-toolbar
22
3---3---
4Optional for postgres usage:4Optional for postgres usage:
55
=== added file 'harvest/common/url_tools.py'
--- harvest/common/url_tools.py 1970-01-01 00:00:00 +0000
+++ harvest/common/url_tools.py 2010-06-27 21:07:25 +0000
@@ -0,0 +1,32 @@
1def new_url_with_parameters(url, params_dict):
2 """
3 Returns a new URL with an added query string, described by params_dict.
4 @param params_dict: a dictionary with all the parameters to add
5 @param path: url to add the parameters to
6 @return: the url (a string) with given parameters
7 """
8 #Derived from <http://djangosnippets.org/snippets/1627/>
9
10 def param_bit(key, value):
11 if value:
12 return "%s=%s" % (key, value)
13 else:
14 return "%s" % key
15
16 if len(params_dict):
17 url_params = list()
18 url += "?%s" % "&".join([param_bit(key, value) for (key, value) in params_dict.items()])
19
20 return url
21
22def current_url_with_parameters(request, new_params_dict):
23 """
24 Returns the current URL with some parameters changed, which are
25 described in new_params_dict. The rest of the query string remains
26 intact.
27 """
28 params = request.GET.copy() #this includes parameters that aren't used by the FilterSystem
29 params.update(new_params_dict)
30 url = request.path
31 return new_url_with_parameters(url, params)
32
033
=== added directory 'harvest/filters'
=== added file 'harvest/filters/__init__.py'
--- harvest/filters/__init__.py 1970-01-01 00:00:00 +0000
+++ harvest/filters/__init__.py 2010-06-27 21:07:25 +0000
@@ -0,0 +1,23 @@
1"""
2Here, we define a very abstract filtering system.
3This system is entirely decoupled from presentation. A filter is simply a query
4operation intended for a particular type of model, which can be configured and
5turned on / off via structured input.
6
7There are, of course, some rules: different types of filters can be added,
8some of which store extra information (like a line of text that the user can
9input), and filters can be grouped like radio buttons where only one filter
10in a group can be selected.
11
12The included classes provide most important features for free, but to be of
13any use they need to be extended to fit the intended application. For example,
14the get_query class is unimplemented; it should return a Q object that does
15the work of selecting whatever the filter is intended to select.
16
17Groups of filters in no way refer to how they get placed in a page, or how
18they are interacted with. We leave that entirely up to templates. It is also
19something else's job to figure out how to draw filters, and how to form the
20links they talk to us with.
21
22All This Does is filter things.
23"""
024
=== added file 'harvest/filters/containers.py'
--- harvest/filters/containers.py 1970-01-01 00:00:00 +0000
+++ harvest/filters/containers.py 2010-06-27 21:07:25 +0000
@@ -0,0 +1,97 @@
1from harvest.common.url_tools import current_url_with_parameters
2
3class FilterContainer(object): #abstract
4 """
5 A class that contains Filter objects, which can be retrieved with the
6 find method.
7
8 The added Filter objects are referred to as "children." They are expected
9 to exist for the entire life of the container.
10 """
11
12 def __init__(self, filters_set):
13 self.filters_dict = dict() #refers to Filter objects by their unique IDs
14 self.add_filters(filters_set)
15
16 def add_filters(self, filters_set): #final
17 """
18 Adds a set of filters to be children of this one and informs
19 each child that it belongs to this container.
20 @param filter_set: a set of Filter objects
21 """
22 for child in set(filters_set):
23 self.filters_dict[child.get_id()] = child
24 child.set_container(self)
25
26 def find(self, full_name): #final
27 """
28 Finds a filter inside this object or one of its children, based
29 on that filter's full name in the format container:child.
30 @param full_name: an object's full name
31 @return: the object described by full_name, or None
32 """
33 result = None
34 name_bits = full_name.split(':',1)
35
36 #the first bit is the one this instance stores
37 if name_bits[0] in self.filters_dict:
38 result = self.filters_dict[name_bits[0]]
39
40 if isinstance(result, FilterContainer) and len(name_bits)>1:
41 result = result.find(name_bits[1]) #send remaining bits to child
42
43 #we have reached the end of the list. Return the result (which may be None)
44 return result
45
46
47
48class FilterSystem(FilterContainer):
49 """
50 This is the single all-knowing root object that should contain all
51 other filters for an application. From this object it is possible to
52 find a filter using its full name, which is the same name used to
53 serialize a filter's state in an HTTP query string.
54
55 Before an instance of this object can be safely used,
56 update_from_http should be called with the current HttpRequest.
57 """
58
59 def __init__(self, filters_set, default_parameters = dict()):
60 FilterContainer.__init__(self, filters_set)
61 self.request = None #current http request
62 self.default_parameters = default_parameters
63
64 def update_from_http(self, request): #final
65 """
66 Call this method to update the state of all filters based on an
67 HttpRequest object. The request object will be stored for other uses.
68 @param request: current HttpRequest object
69 """
70 self.request = request
71 #this contains parameters that aren't relevant to the FilterSystem
72 self._set_parameters(request.GET)
73
74 def _set_parameters(self, parameters): #final
75 """
76 Updates the state of all filters based on given parameters.
77 @param parameters: dictionary of parameters, for example from request.GET
78 """
79 new_params = parameters.copy()
80 for key in self.default_parameters:
81 #apply default parameters for keys that have not been set
82 if not key in new_params: new_params[key] = self.default_parameters[key]
83
84 for key in new_params:
85 filter_object = self.find(key)
86 if filter_object:
87 filter_object.set_value(new_params[key])
88
89 def get_url_with_parameters(self, parameters): #final
90 """
91 Returns a new URL where the given parameters will be applied.
92 To generate parameters, see Filter.serialize.
93 @param parameters: a dictionary of new parameters
94 @return: the current url with the given parameters added to the query string
95 """
96 return current_url_with_parameters(self.request, parameters)
97
098
=== added file 'harvest/filters/filters.py'
--- harvest/filters/filters.py 1970-01-01 00:00:00 +0000
+++ harvest/filters/filters.py 2010-06-27 21:07:25 +0000
@@ -0,0 +1,318 @@
1#FIXME: Make ChoiceFilter a bit nicer so we don't need to call super() and all that bother from harvest.opportunities.filters.
2#TODO: adjust Filter.render() methods for custom template tags, with django.template.Template
3
4from containers import FilterContainer, FilterSystem
5from django.utils.safestring import mark_safe
6from copy import copy
7
8class Filter(object): #abstract, extend in application
9 """
10 The abstract base class for all other filters.
11
12 Every Filter's main objective is to process QuerySets with the method
13 Filter.process_queryset. Filter, or one of its subclasses, should be
14 extended with a new version of process_queryset that does something useful
15 for a specific application. For example, process_queryset could return
16 queryset.filter(foo="bar") so using the filter will limit the queryset
17 to objects where foo=bar.
18
19 Every Filter has a unique id and can belong to a single container.
20
21 A Filter can be assigned a new value from a string and its internal value
22 (which could be of any type) can be serialized back to a string.
23 The object itself can be serialized as a key / value pair, where the key
24 is completely unique.
25
26 Every Filter can be rendered, which outputs markup (currently html)
27 describing it for a user interface.
28 """
29
30 #TODO: figure out get_system and get_full_name when set_container is called and store the value
31
32 def __init__(self, id_str): #final
33 self.id_str = id_str #local name
34 self.container = None #immediate container (FilterContainer)
35
36 def get_id(self): #final
37 """
38 @return: the local id of this filter, unique to its container.
39 """
40 return self.id_str
41
42 def set_container(self, container): #final
43 """
44 Specify that this filter belongs to a specific container. This will
45 replace any container it currently believes it belongs to.
46 @param container: a FilterContainer object that holds this Filter
47 """
48 #it would make sense to raise an exception if self.container != None
49 self.container = container
50
51 def get_container(self): #final
52 """
53 @return: the container that this filter belongs to
54 """
55 return self.container
56
57 def get_system(self): #final
58 """
59 @return: the FilterSystem that this filter ultimately belongs to
60 """
61 container = self.get_container()
62 system = None
63 if isinstance(container, Filter):
64 system = container.get_system()
65 elif isinstance(container, FilterSystem):
66 system = container
67 return system
68
69 def get_full_name(self): #final
70 """
71 Returns the filter's full name, which should make sense from anywhere
72 in the application. This name is in the format parent:child:child.
73 @return: the filter's full name, which is a string
74 """
75 full_name = self.get_id()
76 container = self.get_container()
77 if isinstance(container, Filter):
78 full_name = "%s:%s" % (container.get_full_name(), full_name)
79 return full_name
80
81 def set_value(self, value): #abstract
82 """
83 Extend this to take a value passed down from the top, probably
84 from FilterSystem.update_from_http, and do something with it.
85 @param value: a new value, as a string
86 """
87 raise NotImplementedError(self.set_value)
88
89 def get_value(self): #abstract
90 """
91 @return: a copy of this filter's value in its native format
92 """
93 raise NotImplementedError(self.get_value)
94
95 def serialize_value(self, value): #abstract
96 """
97 The inverse of set_value. Returns the given value as a string that could
98 be added to an HTTP query string.
99 @param value: the value to serialize, in a format native to this Filter (see get_value)
100 @return: a unicode string formatted for set_value
101 """
102 raise NotImplementedError(self.serialize_value)
103
104 def serialize(self, value = None): #final
105 """
106 Creates a dictionary of parameters to describe this object, either
107 as-is or with a new value.
108 The result can be sent to FilterSystem.get_url_with_parameters.
109 @param value: a different value to use, in a format native to this Filter (see get_value)
110 @return: a dictionary of key:value pairs referring to the object and its value.
111 """
112 if value == None: value = self.get_value()
113 key = self.get_full_name()
114 value_str = self.serialize_value(value)
115 return {key : value_str}
116
117 def get_container_toggle_parameters(self): #final
118 """
119 Helper method to get the parameter for toggling this filter's state in
120 its container, if there is one.
121 @return: a dictionary of key:value pairs to generate new GET parameters
122 """
123 container = self.get_container()
124 params = dict()
125 if isinstance(container, FilterGroup):
126 params = container.serialize(container.get_value_with_selection(self.get_id()))
127 return params
128
129 def process_queryset(self, queryset): #abstract
130 """
131 Extend this to manipulate a given queryset and then return it.
132 For example, queryset.filter(name__startswith = self.value)
133 @param queryset: a queryset to operate on
134 @return: a queryset based on the given one
135 """
136 raise NotImplementedError(self.process_queryset)
137
138 def render(self): #final
139 """
140 @return: the default rendering of the filter itself in given context
141 """
142 return self.render_html()
143
144 def render_html(self):
145 """
146 Extend this to return the html output for the filter itself.
147 The output should be very simple and semantically meaningful,
148 with no specific concern about formatting. It will be
149 placed within other tags that describe its context, and it is
150 up to the template to describe style.
151 @return: a unicode string containing html representing this filter
152 """
153 system = self.get_system()
154 toggle_params = self.get_container_toggle_parameters()
155 href = system.get_url_with_parameters(toggle_params)
156
157 return mark_safe(u'<a href="%s">(%s)</a>'
158 % (href, self.get_id()))
159
160
161
162
163class EditFilter(Filter): #abstract, extend in application
164 """
165 This Filter has a simple string value which can be edited by the user.
166
167 Serialized as stored ("value")
168 """
169
170 def __init__(self, id_str):
171 Filter.__init__(self, id_str)
172 self.input_str = ""
173
174 def set_value(self, value): #overrides Filter
175 self.input_str = value
176
177 def get_value(self): #overrides Filter
178 return self.input_str
179
180 def serialize_value(self, value): #overrides Filter
181 return value
182
183 def render_html(self):
184 system = self.get_system()
185 toggle_params = self.get_container_toggle_parameters()
186 href = system.get_url_with_parameters(toggle_params)
187
188 return mark_safe(u'<a href="%s">%s: %s</a>'
189 % (href, self.get_id(), self.get_value()))
190
191
192class SetFilter(Filter): #abstract, extend in application
193 """
194 Holds a set of strings, with no repetition.
195
196 Serialized as a comma-separated list ("dog,cat,horse,mouse")
197 """
198
199 def __init__(self, id_str):
200 Filter.__init__(self, id_str)
201 self.selected_set = set()
202
203 def set_value(self, value): #overrides Filter
204 self.selected_set = set([s for s in value.split(",") if self.id_allowed(s)])
205
206 def get_value(self): #overrides Filter
207 return self.selected_set.copy()
208
209 def serialize_value(self, value): #overrides Filter
210 return ",".join(value)
211
212 def get_value_with_selection(self, item_id): #final
213 """
214 Returns the current value of this object with the selection referred to
215 by item_id toggled on or off, depending on its current state.
216 @param item_id: id for the item to toggle
217 @return: the value of this SetFilter with the given item toggled on or off
218 """
219 select = self.get_value()
220 if item_id in select:
221 select.remove(item_id)
222 else:
223 select.add(item_id)
224 return select
225
226 def id_selected(self, item_id):
227 return item_id in self.selected_set
228
229 def id_allowed(self, item_id):
230 return True
231
232
233class ChoiceFilter(SetFilter): #abstract, extend in application
234 """
235 Has a dictionary of items, with names and values of any type. These can be
236 selected or deselected by the input, which is a set of strings, as in
237 SetFilter. In that set, any items which refer to choices that do not exist
238 are ignored.
239
240 Serialized as a comma-separated list, like SetFilter.
241 """
242
243 def __init__(self, id_str, choices_dict):
244 SetFilter.__init__(self, id_str)
245 self.choices_dict = choices_dict
246
247 def id_allowed(self, item_id): #overrides SetFilter
248 return item_id in self.choices_dict
249
250 def get_selected_items(self):
251 return [self.choices_dict[s] for s in self.selected_set]
252
253 def render_html(self): #overrides Filter
254 choices = ""
255
256 for c in self.choices_dict:
257 c_render = self._render_html_choice(c)
258 if self.id_selected(c):
259 c_render = "<b>%s</b>" % c_render
260 choices += "<li>%s</li>" % c_render
261
262 system = self.get_system()
263 toggle_params = self.get_container_toggle_parameters()
264 self_href = system.get_url_with_parameters(toggle_params)
265
266 return mark_safe(u'<a href="%s">%s</a>:<ul>%s</ul>' % (self_href, self.get_id(), choices))
267
268 def _render_html_choice(self, item_id):
269 system = self.get_system()
270 toggle_params = self.serialize(self.get_value_with_selection(item_id))
271 item_href = system.get_url_with_parameters(toggle_params)
272
273 return mark_safe(u'<a href="%s">%s</a>' % (item_href, item_id))
274
275
276
277
278class FilterGroup(FilterContainer, SetFilter): #final
279 """
280 A collection of other Filters, which are selected (enabled) according to the
281 rules of SetFilter.
282
283 The do_queryset method mixes the output from all selected Filters, so only
284 the one for this FilterGroup needs to be (or should be) called.
285
286 Serialized as a comma-separated list, like SetFilter.
287 """
288
289 def __init__(self, id_str, filters_set):
290 FilterContainer.__init__(self, filters_set)
291 SetFilter.__init__(self, id_str)
292
293 def id_allowed(self, item_id): #overrides SetFilter
294 return item_id in self.filters_dict
295
296 def get_selected_filters(self, filter_id):
297 return [self.filters_dict[s] for s in self.selected_set]
298
299 def process_queryset(self, queryset): #overrides Filter
300 for f in self.selected_set:
301 queryset = self.filters_dict[f].process_queryset(queryset) #returns something like QuerySet.filter(blah)
302 return queryset
303
304 def render_html(self): #overrides Filter
305 filters = ""
306
307 for f in self.filters_dict:
308 f_render = self._render_html_filter(f)
309 if self.id_selected(f):
310 f_render = "<em>%s</em>" % f_render
311 filters += "<li>%s</li>" % f_render
312
313 return mark_safe(u'%s:<ul>%s</ul>' % (self.get_id(), filters))
314
315 def _render_html_filter(self, filter_id):
316 f = self.filters_dict[filter_id]
317 return f.render_html()
318
0319
=== added file 'harvest/opportunities/filters.py'
--- harvest/opportunities/filters.py 1970-01-01 00:00:00 +0000
+++ harvest/opportunities/filters.py 2010-06-27 21:07:25 +0000
@@ -0,0 +1,53 @@
1from harvest.filters import filters, containers
2import models
3
4class PkgNameFilter(filters.EditFilter):
5 def process_queryset(self, queryset):
6 return queryset.filter(name__startswith = self.get_value())
7
8class PkgSetFilter(filters.ChoiceFilter):
9 def __init__(self, id_str):
10 choices_dict = dict()
11 for s in models.PackageSet.objects.all():
12 choices_dict[s.name] = s
13 super(PkgSetFilter, self).__init__(id_str, choices_dict)
14
15 def process_queryset(self, queryset):
16 return queryset.filter(packagesets__in=self.get_selected_items())
17
18
19
20class OppFeaturedFilter(filters.Filter):
21 def process_queryset(self, queryset):
22 return queryset.filter(opportunitylist__featured=True)
23
24class OppListFilter(filters.ChoiceFilter):
25 def __init__(self, id_str):
26 choices_dict = dict()
27 for l in models.OpportunityList.objects.all():
28 choices_dict[l.name] = l
29 super(OppListFilter, self).__init__(id_str, choices_dict)
30
31 def process_queryset(self, queryset):
32 return queryset.filter(opportunitylist__in=self.get_selected_items())
33
34
35#we don't really need to create a special type here, but it may be handy
36class HarvestFilters(containers.FilterSystem):
37 def __init__(self):
38 super(HarvestFilters, self).__init__(
39 [
40 filters.FilterGroup("pkg", [
41 PkgNameFilter("name"),
42 PkgSetFilter("set")
43 ] ),
44 filters.FilterGroup("opp", [
45 OppFeaturedFilter("featured"),
46 OppListFilter("list")
47 ] )
48 ],
49 default_parameters = { "pkg" : "name,set",
50 "pkg:name" : "ged",
51 "pkg:set" : "ubuntu-desktop" }
52 )
53
054
=== modified file 'harvest/opportunities/urls.py'
--- harvest/opportunities/urls.py 2010-03-08 16:33:21 +0000
+++ harvest/opportunities/urls.py 2010-06-27 21:07:25 +0000
@@ -12,6 +12,10 @@
1212
13 url(r'^source-package/(?P<sourcepackage_slug>[-\w+.]+)/$', 'opportunities.views.sourcepackage_detail', name='sourcepackage_detail'),13 url(r'^source-package/(?P<sourcepackage_slug>[-\w+.]+)/$', 'opportunities.views.sourcepackage_detail', name='sourcepackage_detail'),
14 url(r'^source-package/$', 'opportunities.views.sourcepackage_list', name='sourcepackage_list'),14 url(r'^source-package/$', 'opportunities.views.sourcepackage_list', name='sourcepackage_list'),
15
16 url(r'^filter',
17 'opportunities.views.opportunities_filter',
18 name='opportunities_filter'),
1519
16 url(r'^by-type',20 url(r'^by-type',
17 'opportunities.views.opportunities_by_type',21 'opportunities.views.opportunities_by_type',
1822
=== modified file 'harvest/opportunities/views.py'
--- harvest/opportunities/views.py 2010-06-08 15:46:42 +0000
+++ harvest/opportunities/views.py 2010-06-27 21:07:25 +0000
@@ -13,6 +13,9 @@
13import models13import models
14import forms14import forms
1515
16from filters import HarvestFilters
17from wrappers import PackageWrapper, PackageListWrapper
18
16def opportunity_index(request):19def opportunity_index(request):
17 sources_list = models.SourcePackage.objects.all()20 sources_list = models.SourcePackage.objects.all()
18 paginator = Paginator(sources_list, 50)21 paginator = Paginator(sources_list, 50)
@@ -120,6 +123,37 @@
120 extra_context = {'opportunities': opportunities},123 extra_context = {'opportunities': opportunities},
121 )124 )
122125
126def opportunities_filter(request):
127 filters = HarvestFilters()
128 filters.update_from_http(request)
129 filters_pkg = filters.find('pkg')
130 filters_opp = filters.find('opp')
131
132 packages_list = models.SourcePackage.objects.distinct()
133 packages_list = filters_pkg.process_queryset(packages_list)
134
135 #opportunities_list is filtered right away to only check opportunities belonging to selected packages
136 opportunities_list = models.Opportunity.objects.distinct().filter(sourcepackage__in=packages_list)
137 opportunities_list = filters_opp.process_queryset(opportunities_list)
138 #TODO: need to filter out opportunities with valid=False again
139 #TODO: would it be more efficient to group opportunities by their sourcepackages first, then run filters_opp.process_queryset() for each of those groups?
140
141 pkg_list_wrapper = PackageListWrapper(request, packages_list, opportunities_list)
142
143 context = {
144 'grouping': 'package',
145 'packages_list': pkg_list_wrapper,
146 'filters_pkg' : filters_pkg,
147 'filters_opp' : filters_opp
148 }
149
150 return render(
151 'opportunities/opportunities_filter.html',
152 context,
153 context_instance=RequestContext(request))
154
155#TODO: package_filter_detail(request, sourcepackage, opportunities_list)
156
123def opportunities_by_type(request):157def opportunities_by_type(request):
124 types_list = models.OpportunityList.objects.filter(active=True)158 types_list = models.OpportunityList.objects.filter(active=True)
125 paginator = Paginator(types_list, 50)159 paginator = Paginator(types_list, 50)
126160
=== added file 'harvest/opportunities/wrappers.py'
--- harvest/opportunities/wrappers.py 1970-01-01 00:00:00 +0000
+++ harvest/opportunities/wrappers.py 2010-06-27 21:07:25 +0000
@@ -0,0 +1,90 @@
1from django.db.models import Count
2from harvest.common.url_tools import current_url_with_parameters
3
4class PackageWrapper(object):
5 """
6 Describes a visible source package, for specific use in a
7 template.
8 """
9
10 def __init__(self, request, package, visible_opportunities = None, expanded = False):
11 self.request = request
12 self.package = package
13 self.visible_opportunities = visible_opportunities
14 self.expanded = expanded
15
16 def real(self):
17 return self.package
18
19 def get_expand_toggle_url(self):
20 parameter = {'expand_pkg' : self.package.name}
21 url = current_url_with_parameters(self.request, parameter)
22 return url
23
24 #FIXME: get_visible_opportunities and get_hidden_opportunities feel
25 # wasteful. Could we do exclude and filter in a single
26 # operation? Does it affect performance?
27 def get_visible_opportunities(self):
28 """
29 Returns opportunities that belong to the given package and are
30 in opportunities_list.
31 """
32 #also check if valid?
33 return self.visible_opportunities
34
35 def get_hidden_opportunities(self):
36 """
37 Returns opportunities that belong to the given package but have
38 been hidden from view
39 """
40 opps_visible = self.get_visible_opportunities()
41 return self.package.opportunity_set.exclude(pk__in=opps_visible)
42
43class PackageListWrapper(object):
44 """
45 Object describing a list of source packages and opportunities, to
46 be used by a template. It contains UI-specific variables and simple
47 helper functions for doing final queries to access these lists.
48 """
49
50 def __init__(self, request, packages_list, opportunities_list):
51 expand_list = None #list of packages to show in detail
52 if 'expand_pkg' in request.GET:
53 expand_list = request.GET['expand_pkg'].split(',')
54
55 related_packages = set(opportunities_list.values_list('sourcepackage', flat=True))
56
57 self.visible_packages_list = list()
58 self.hidden_packages_list = list()
59
60 #Create a PackageWrapper around every source package.
61 #Includes a less detailed wrapper for hidden packages.
62 for package in packages_list:
63 if package.pk in related_packages:
64 opps = None
65 expand = False
66
67 if expand_list: expand = (package.name in expand_list)
68 opps = opportunities_list.filter(sourcepackage=package)
69
70 package_wrapper = PackageWrapper(request, package,
71 visible_opportunities = opps,
72 expanded = expand)
73 self.visible_packages_list.append(package_wrapper)
74
75 else:
76 package_wrapper = PackageWrapper(request, package)
77 self.hidden_packages_list.append(package_wrapper)
78
79 def get_visible_packages(self):
80 """
81 Returns list of packages that are are visible.
82 These are any packages that contain opportunities.
83 """
84 return self.visible_packages_list
85
86 def get_hidden_packages(self):
87 """
88 Returns list of packages that have been hidden from view.
89 """
90 return self.hidden_packages_list
0\ No newline at end of file91\ No newline at end of file
192
=== modified file 'harvest/settings.py'
--- harvest/settings.py 2010-06-01 16:16:19 +0000
+++ harvest/settings.py 2010-06-27 21:07:25 +0000
@@ -7,6 +7,7 @@
7TEMPLATE_DEBUG = DEBUG7TEMPLATE_DEBUG = DEBUG
8STATIC_SERVE = True8STATIC_SERVE = True
9PROJECT_NAME = 'harvest'9PROJECT_NAME = 'harvest'
10INTERNAL_IPS = ('127.0.0.1',) #for testing
1011
11from common import utils12from common import utils
12VERSION_STRING = utils.get_harvest_version(13VERSION_STRING = utils.get_harvest_version(
@@ -70,6 +71,7 @@
70 'django.contrib.sessions.middleware.SessionMiddleware',71 'django.contrib.sessions.middleware.SessionMiddleware',
71 'django.contrib.auth.middleware.AuthenticationMiddleware',72 'django.contrib.auth.middleware.AuthenticationMiddleware',
72 'django.middleware.locale.LocaleMiddleware',73 'django.middleware.locale.LocaleMiddleware',
74 'debug_toolbar.middleware.DebugToolbarMiddleware', #for testing
73)75)
7476
75ROOT_URLCONF = 'harvest.urls'77ROOT_URLCONF = 'harvest.urls'
@@ -98,7 +100,9 @@
98 'django.contrib.sites',100 'django.contrib.sites',
99 'django.contrib.admin',101 'django.contrib.admin',
100 'django_openid_auth',102 'django_openid_auth',
103 'debug_toolbar', #for testing
101 'opportunities',104 'opportunities',
105 'filters',
102 'common',106 'common',
103)107)
104108
105109
=== added file 'harvest/templates/opportunities/opportunities_filter.html'
--- harvest/templates/opportunities/opportunities_filter.html 1970-01-01 00:00:00 +0000
+++ harvest/templates/opportunities/opportunities_filter.html 2010-06-27 21:07:25 +0000
@@ -0,0 +1,50 @@
1{% extends "base.html" %}
2{% load i18n %}
3
4{% block title %}{% trans "Opportunity Index" %} - {{ block.super }}{% endblock %}
5
6{% block content %}
7<div class="mainpage">
8
9<h1>{% trans "Opportunities" %}</h1>
10
11<div class="filters" style="background-color:#E0F1FF; float:left; width:15em;">
12 {{filters_pkg.render}}
13 {{filters_opp.render}}
14</div>
15
16<div class="results" style="float:left;">
17{% if packages_list %}
18<ul>
19 {% for pkg in packages_list.get_visible_packages %}
20 <li><a href="{{ pkg.get_expand_toggle_url }}">{{ pkg.real.name }}</a>
21 {% if pkg.expanded %}
22 <ul>
23 {% for opportunity in pkg.get_visible_opportunities %}
24 {% include "opportunities/opportunity_detail.inc.html" %}
25 {% endfor %}
26
27 {% with pkg.get_hidden_opportunities.count as hidden_count %}
28 {% ifnotequal hidden_count 0 %}
29 <li><small>{{ hidden_count }} {{ hidden_count|pluralize:"opportunity,opportunities"}} hidden</small></li>
30 {% endifnotequal %}
31 {% endwith %}
32 </ul>
33 {% endif %}
34 </li>
35 {% endfor %}
36
37 {% with packages_list.get_hidden_packages|length as hidden_count %}
38 {% ifnotequal hidden_count 0 %}
39 <li><small>{{ hidden_count }} package{{ hidden_count|pluralize:"s"}} {{ hidden_count|pluralize:"has,have"}} no matching opportunities</small></li>
40 {% endifnotequal %}
41 {% endwith %}
42</ul>
43
44{% else %}
45<p>{% trans "There are currently no opportunities in Harvest. :(" %}</p>
46{% endif %}
47</div>
48
49</div>
50{% endblock %}

Subscribers

People subscribed via source and target branches

to all changes: