Storm

Merge lp:~iffy/storm/lazycolumns into lp:storm

  1. lazycolumns
  2. Merge into trunk
Proposed by Matt Haggard
Status: Needs review
Proposed branch: lp:~iffy/storm/lazycolumns
Merge into: lp:storm
Diff against target: 516 lines (+203/-44)
10 files modified
TODO (+0/-26)
storm/expr.py (+8/-2)
storm/info.py (+22/-0)
storm/properties.py (+20/-6)
storm/store.py (+37/-8)
tests/info.py (+35/-1)
tests/store/base.py (+70/-1)
tests/store/mysql.py (+5/-0)
tests/store/postgres.py (+3/-0)
tests/store/sqlite.py (+3/-0)
To merge this branch: bzr merge lp:~iffy/storm/lazycolumns
Reviewer Review Type Date Requested Status
Storm Developers Pending
Review via email: mp+87092@code.launchpad.net

Description of the change

Implemented the lazy-loading feature as described in the TODO file. Instead of having separate "lazy" and "lazy_group" arguments, however, I opted for just requiring "lazy" which can be one of the following:
 - True: The column should be lazy all by itself.
 - True-ish values: The column should be grouped with all other columns using this same value. If any of them are accessed, all of them are loaded.

Rationale: We need this to speed up access of our bad tables with huge text fields that are rarely used.

To post a comment you must log in.
Revision history for this message
Robert Collins (lifeless) wrote :

I'm glad you are working on this.

I have not done a full review at this point; I'd like to clarify
something though - it looks like you are making it so that all queries
will assume these columns are lazy. It is more efficient to grab the
column if needed in a single query rather than one-per-object. It
would be nice if users had the ability to control that.

For instance, say you have a changelog field in a table, which is a
big text field. Most pages in the website may not need it, but some
do. If its marked lazy, the pages that do need it will end up doing
(number of changelogs shown) separate queries to lazy-load the
changelog. If there is someway to say 'for this query, the changelog
field is needed', then that overhead can be eliminated.

And, if we have such a way to force laziness-into-eagerness, we
probably can do the reverse trivially - force eagerness into laziness
to prevent a bunch of fields being loaded on a query by query basis.

What do you think?

-Rob

Reply
Revision history for this message
Matt Haggard (iffy) wrote :

Rob,

I know what you mean. I'm working on this to fix a problem we currently have with a big table (126 fields). 90% of the time, we need about 4 fields; 10% of the time we need some combination of the others.

My current patch allows you to define column laziness globally, which improves 90% of our code. What you're suggesting is that in addition to (instead of?) defining laziness globally, you can override laziness when using a store.find, store.get, Reference or ReferenceSet? This would allow one to avoid double queries for the other 10%.

What would the syntax look like?

store.find(Foo, eager=(Foo.name, Foo.kind))
bar = Reference(bar_id, 'Bar.id', eager=('Bar.name', 'Bar.kind'))

Or would you do some kind of state thing (this tastes bad)?

Bar.eagers.push('name', 'kind')
store.find(Bar)
Bar.eagers.pop()

Maybe this is better:

store.eager_load(Bar.name, Bar.kind)
store.find(Bar)
...
store.lazy_load(Bar.name, Bar.kind)

Also, is laziness overriding (your suggestion) something that should be built in from the beginning, or can it be added as a new feature after global laziness (my patch) is added? Maybe the answer depends on the implementation.

Thanks,

Matt

Reply
Revision history for this message
Jamu Kakar (jkakar) wrote :

There are some notes in the TODO file that you might draw inspiration
from.

Reply
Revision history for this message
Matt Haggard (iffy) wrote :

> There are some notes in the TODO file that you might draw inspiration
> from.

Yes, those are great notes! This branch implements laziness as described there.

Reply
Revision history for this message
Robert Collins (lifeless) wrote :

On Wed, Jan 4, 2012 at 5:33 AM, Matt Haggard <email address hidden> wrote:
> Rob,
> Also, is laziness overriding (your suggestion) something that should be built in from the beginning, or can it be added as a new feature after global laziness (my patch) is added?  Maybe the answer depends on the implementation.

I think adding it later is fine, but we should consider what it needs
early, so that we don't conflict with the global laziness thing, which
will itself be useful.

As for syntax, perhaps
# do not load attribute1 or attribute2, and load attribute3 which is
globally set to lazy
store.find(Load(Foo, lazy=['attribute1', 'attribute2'], eager=['attribute3'])) ?

Reply
Revision history for this message
Matt Haggard (iffy) wrote :

So what's the next step on this?

Reply
Revision history for this message
Robert Collins (lifeless) wrote :

On Fri, Jan 6, 2012 at 11:19 AM, Matt Haggard <email address hidden> wrote:
> So what's the next step on this?

Well, I think for you to spend a little time coming up with an answer to
'does the patch need to change to gracefully permit per-field laziness
in future'

if the answer is no, then we review and land; if it is yes, you make
such changes as you consider necessary to future proof, then we review
and land.

-Rob

Reply
Revision history for this message
Matt Haggard (iffy) wrote :

Rob,

No, this patch doesn't need to change to support per-query laziness in the future. Per-query laziness would use some of the stuff in this patch as it is. The overlap point is probably the LazyGroupMember class, which would probably be changed to Unloaded, NotLoadedYet, NotFetchedYet or Unfetched or something like that. But that's used internally and not in the public API.

Thanks for your help on this!

Matt

Reply
Revision history for this message
Gustavo Niemeyer (niemeyer) wrote :

Thanks a lot for pushing this forward.

Given the question on how the syntax should look like, I'm concerned the
spirit described in the notes isn't being implemented. It doesn't seem
practical to define per field granularity for what should be loaded.

I'll have a look at the branch, hopefully next week.
On Jan 3, 2012 2:33 PM, "Matt Haggard" <email address hidden> wrote:

> Rob,
>
> I know what you mean. I'm working on this to fix a problem we currently
> have with a big table (126 fields). 90% of the time, we need about 4
> fields; 10% of the time we need some combination of the others.
>
> My current patch allows you to define column laziness globally, which
> improves 90% of our code. What you're suggesting is that in addition to
> (instead of?) defining laziness globally, you can override laziness when
> using a store.find, store.get, Reference or ReferenceSet? This would allow
> one to avoid double queries for the other 10%.
>
> What would the syntax look like?
>
> store.find(Foo, eager=(Foo.name, Foo.kind))
> bar = Reference(bar_id, 'Bar.id', eager=('Bar.name', 'Bar.kind'))
>
> Or would you do some kind of state thing (this tastes bad)?
>
> Bar.eagers.push('name', 'kind')
> store.find(Bar)
> Bar.eagers.pop()
>
> Maybe this is better:
>
> store.eager_load(Bar.name, Bar.kind)
> store.find(Bar)
> ...
> store.lazy_load(Bar.name, Bar.kind)
>
> Also, is laziness overriding (your suggestion) something that should be
> built in from the beginning, or can it be added as a new feature after
> global laziness (my patch) is added? Maybe the answer depends on the
> implementation.
>
>
> Thanks,
>
> Matt
> --
> https://code.launchpad.net/~magmatt/storm/lazycolumns/+merge/87092
> You are subscribed to branch lp:storm.
>

Reply
Revision history for this message
Matt Haggard (iffy) wrote :

Gustavo,

I'm excited for your review.

To clarify, this branch implements laziness as described in the TODO. Rob mentioned that it would be nice to sometimes override that behavior per query. That (overriding per query) is not implemented here. So the syntax closely mirrors the TODO.

A single, lazy attr:

    class C(object):
        ...
        attr = Unicode(lazy=True)

or a group of lazy attrs (loaded together if either is accessed):

    class C(object):
        ...
        attr = Unicode(lazy=2)
        attr2 = Unicode(lazy=2)

    class D(object):
        ...
        attr = Int(lazy='group 1')
        attr2 = Int(lazy='group 2')

^ Is the syntax supported by this branch. I opted for a single "lazy" arg instead of having a "lazy" and "lazy_group." It seems simpler to me.

Reply

Unmerged revisions

431. By Matt Haggard

Added docstring for lazy attribute in expr.Column and removed laziness from the
TODO file

430. By Matt Haggard

Lazy-loaded columns will not overwrite a previously set value.

429. By Matt Haggard

Finished lazy-loading of columns including some Store tests.

428. By Matt Haggard

Added lazy keyword as __init__ to Property and changed ClassInfo to
gather appropriate laziness information from that.

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== modified file 'TODO'
--- TODO 2008-10-29 15:49:19 +0000
+++ TODO 2011-12-29 16:23:23 +0000
@@ -7,32 +7,6 @@
77
8- Unicode(autoreload=True) will mark the field as autoreload by default.8- Unicode(autoreload=True) will mark the field as autoreload by default.
99
10- Lazy-by-default attributes:
11
12 class C(object):
13 ...
14 attr = Unicode(lazy=True)
15
16 This would make attr be loaded only if touched.
17
18 Or maybe lazy groups:
19
20 class C(object):
21 ...
22 lazy_group = LazyGroup()
23 attr = Unicode(lazy=True, lazy_group=lazy_group)
24
25 Once one of the group attributes are accessed all of them are retrieved
26 at the same time.
27
28 Lazy groups may be integers as well:
29
30 class C(object):
31 ...
32 attr = Unicode(lazy_group=1)
33
34 lazy_group=None means not lazy.
35
36- Implement ResultSet.reverse[d]() to invert order_by()?10- Implement ResultSet.reverse[d]() to invert order_by()?
3711
38- Add support to cyclic references when all of elements of the cycle are12- Add support to cyclic references when all of elements of the cycle are
3913
=== modified file 'storm/expr.py'
--- storm/expr.py 2011-09-15 13:07:55 +0000
+++ storm/expr.py 2011-12-29 16:23:23 +0000
@@ -799,15 +799,21 @@
799 a bool.799 a bool.
800 @ivar variable_factory: Factory producing C{Variable} instances typed800 @ivar variable_factory: Factory producing C{Variable} instances typed
801 according to this column.801 according to this column.
802 @ivar lazy: Anything trueish indicates that this column should not be
803 loaded from the database when an object is fetched. If this is
804 C{True} the column will be loaded on first access. If the value is
805 an integer or string, then all columns with the same value will be
806 loaded when any of them is first accessed.
802 """807 """
803 __slots__ = ("name", "table", "primary", "variable_factory",808 __slots__ = ("name", "table", "primary", "lazy", "variable_factory",
804 "compile_cache", "compile_id")809 "compile_cache", "compile_id")
805810
806 def __init__(self, name=Undef, table=Undef, primary=False,811 def __init__(self, name=Undef, table=Undef, primary=False,
807 variable_factory=None):812 variable_factory=None, lazy=None):
808 self.name = name813 self.name = name
809 self.table = table814 self.table = table
810 self.primary = int(primary)815 self.primary = int(primary)
816 self.lazy = lazy
811 self.variable_factory = variable_factory or Variable817 self.variable_factory = variable_factory or Variable
812 self.compile_cache = None818 self.compile_cache = None
813 self.compile_id = None819 self.compile_id = None
814820
=== modified file 'storm/info.py'
--- storm/info.py 2011-08-14 08:55:15 +0000
+++ storm/info.py 2011-12-29 16:23:23 +0000
@@ -62,11 +62,16 @@
62 @ivar table: Expression from where columns will be looked up.62 @ivar table: Expression from where columns will be looked up.
63 @ivar cls: Class which should be used to build objects.63 @ivar cls: Class which should be used to build objects.
64 @ivar columns: Tuple of column properties found in the class.64 @ivar columns: Tuple of column properties found in the class.
65 @ivar eager_columns: Tuple of columns that are not lazy.
66 @ivar lazy_columns: Tuple of columns that are loaded lazily.
67 @ivar lazy_groups: Dictionary whose keys are columns and values are lists
68 of columns that should be lazily loaded together.
65 @ivar primary_key: Tuple of column properties used to form the primary key69 @ivar primary_key: Tuple of column properties used to form the primary key
66 @ivar primary_key_pos: Position of primary_key items in the columns tuple.70 @ivar primary_key_pos: Position of primary_key items in the columns tuple.
67 """71 """
6872
69 def __init__(self, cls):73 def __init__(self, cls):
74 from storm.properties import Group
70 self.table = getattr(cls, "__storm_table__", None)75 self.table = getattr(cls, "__storm_table__", None)
71 if self.table is None:76 if self.table is None:
72 raise ClassInfoError("%s.__storm_table__ missing" % repr(cls))77 raise ClassInfoError("%s.__storm_table__ missing" % repr(cls))
@@ -81,11 +86,28 @@
81 column = getattr(cls, attr, None)86 column = getattr(cls, attr, None)
82 if isinstance(column, Column):87 if isinstance(column, Column):
83 pairs.append((attr, column))88 pairs.append((attr, column))
89
8490
85 pairs.sort()91 pairs.sort()
8692
87 self.columns = tuple(pair[1] for pair in pairs)93 self.columns = tuple(pair[1] for pair in pairs)
88 self.attributes = dict(pairs)94 self.attributes = dict(pairs)
95 self.eager_columns = tuple(pair[1] for pair in pairs
96 if not(pair[1].lazy))
97 self.lazy_columns = tuple(pair[1] for pair in pairs
98 if pair[1].lazy)
99
100 common_laziness = {}
101 self.lazy_groups = {}
102 for column in self.lazy_columns:
103 if column.lazy is True:
104 # lazy alone
105 self.lazy_groups[column] = [column]
106 else:
107 # lazy, perhaps with some other columns
108 common = common_laziness.setdefault(column.lazy, [])
109 common.append(column)
110 self.lazy_groups[column] = common
89111
90 storm_primary = getattr(cls, "__storm_primary__", None)112 storm_primary = getattr(cls, "__storm_primary__", None)
91 if storm_primary is not None:113 if storm_primary is not None:
92114
=== modified file 'storm/properties.py'
--- storm/properties.py 2011-02-28 21:16:29 +0000
+++ storm/properties.py 2011-12-29 16:23:23 +0000
@@ -36,17 +36,18 @@
36__all__ = ["Property", "SimpleProperty",36__all__ = ["Property", "SimpleProperty",
37 "Bool", "Int", "Float", "Decimal", "RawStr", "Unicode",37 "Bool", "Int", "Float", "Decimal", "RawStr", "Unicode",
38 "DateTime", "Date", "Time", "TimeDelta", "UUID", "Enum",38 "DateTime", "Date", "Time", "TimeDelta", "UUID", "Enum",
39 "Pickle", "JSON", "List", "PropertyRegistry"]39 "Pickle", "JSON", "List", "PropertyRegistry", "Group"]
4040
4141
42class Property(object):42class Property(object):
4343
44 def __init__(self, name=None, primary=False,44 def __init__(self, name=None, primary=False,
45 variable_class=Variable, variable_kwargs={}):45 variable_class=Variable, variable_kwargs={}, lazy=None):
46 self._name = name46 self._name = name
47 self._primary = primary47 self._primary = primary
48 self._variable_class = variable_class48 self._variable_class = variable_class
49 self._variable_kwargs = variable_kwargs49 self._variable_kwargs = variable_kwargs
50 self._lazy = lazy
5051
51 def __get__(self, obj, cls=None):52 def __get__(self, obj, cls=None):
52 if obj is None:53 if obj is None:
@@ -100,6 +101,7 @@
100 else:101 else:
101 name = self._name102 name = self._name
102 column = PropertyColumn(self, cls, attr, name, self._primary,103 column = PropertyColumn(self, cls, attr, name, self._primary,
104 self._lazy,
103 self._variable_class,105 self._variable_class,
104 self._variable_kwargs)106 self._variable_kwargs)
105 cls._storm_columns[self] = column107 cls._storm_columns[self] = column
@@ -108,12 +110,13 @@
108110
109class PropertyColumn(Column):111class PropertyColumn(Column):
110112
111 def __init__(self, prop, cls, attr, name, primary,113 def __init__(self, prop, cls, attr, name, primary, lazy,
112 variable_class, variable_kwargs):114 variable_class, variable_kwargs):
113 Column.__init__(self, name, cls, primary,115 Column.__init__(self, name, cls, primary,
114 VariableFactory(variable_class, column=self,116 VariableFactory(variable_class, column=self,
115 validator_attribute=attr,117 validator_attribute=attr,
116 **variable_kwargs))118 **variable_kwargs),
119 lazy)
117120
118 self.cls = cls # Used by references121 self.cls = cls # Used by references
119122
@@ -127,10 +130,11 @@
127130
128 variable_class = None131 variable_class = None
129132
130 def __init__(self, name=None, primary=False, **kwargs):133 def __init__(self, name=None, primary=False, lazy=None, **kwargs):
131 kwargs["value"] = kwargs.pop("default", Undef)134 kwargs["value"] = kwargs.pop("default", Undef)
132 kwargs["value_factory"] = kwargs.pop("default_factory", Undef)135 kwargs["value_factory"] = kwargs.pop("default_factory", Undef)
133 Property.__init__(self, name, primary, self.variable_class, kwargs)136 Property.__init__(self, name, primary, self.variable_class, kwargs,
137 lazy)
134138
135139
136class Bool(SimpleProperty):140class Bool(SimpleProperty):
@@ -340,3 +344,13 @@
340 self._storm_property_registry = PropertyRegistry()344 self._storm_property_registry = PropertyRegistry()
341 elif hasattr(self, "__storm_table__"):345 elif hasattr(self, "__storm_table__"):
342 self._storm_property_registry.add_class(self)346 self._storm_property_registry.add_class(self)
347
348
349
350class Group(object):
351 """I am used to group L{Property}s to be lazily loaded together.
352 """
353
354
355
356
343357
=== modified file 'storm/store.py'
--- storm/store.py 2011-05-16 10:45:52 +0000
+++ storm/store.py 2011-12-29 16:23:23 +0000
@@ -42,7 +42,7 @@
42from storm.event import EventSystem42from storm.event import EventSystem
4343
4444
45__all__ = ["Store", "AutoReload", "EmptyResultSet"]45__all__ = ["Store", "AutoReload", "EmptyResultSet", "LazyGroupMember"]
4646
4747
48PENDING_ADD = 148PENDING_ADD = 1
@@ -172,7 +172,7 @@
172172
173 where = compare_columns(cls_info.primary_key, primary_vars)173 where = compare_columns(cls_info.primary_key, primary_vars)
174174
175 select = Select(cls_info.columns, where,175 select = Select(cls_info.eager_columns, where,
176 default_tables=cls_info.table, limit=1)176 default_tables=cls_info.table, limit=1)
177177
178 result = self._connection.execute(select)178 result = self._connection.execute(select)
@@ -667,7 +667,7 @@
667667
668 # Prepare cache key.668 # Prepare cache key.
669 primary_vars = []669 primary_vars = []
670 columns = cls_info.columns670 columns = cls_info.eager_columns + cls_info.lazy_columns
671671
672 for value in values:672 for value in values:
673 if value is not None:673 if value is not None:
@@ -677,6 +677,8 @@
677 # wasn't found. This is useful for joins, where non-existent677 # wasn't found. This is useful for joins, where non-existent
678 # rows are represented like that.678 # rows are represented like that.
679 return None679 return None
680
681 values += tuple([LazyGroupMember] * len(cls_info.lazy_columns))
680682
681 for i in cls_info.primary_key_pos:683 for i in cls_info.primary_key_pos:
682 value = values[i]684 value = values[i]
@@ -694,7 +696,7 @@
694696
695 # Take that chance and fill up any undefined variables697 # Take that chance and fill up any undefined variables
696 # with fresh data, since we got it anyway.698 # with fresh data, since we got it anyway.
697 self._set_values(obj_info, cls_info.columns, result,699 self._set_values(obj_info, columns, result,
698 values, keep_defined=True)700 values, keep_defined=True)
699701
700 # We're not sure if the obj is still in memory at this702 # We're not sure if the obj is still in memory at this
@@ -707,7 +709,7 @@
707 obj_info = get_obj_info(obj)709 obj_info = get_obj_info(obj)
708 obj_info["store"] = self710 obj_info["store"] = self
709711
710 self._set_values(obj_info, cls_info.columns, result, values,712 self._set_values(obj_info, columns, result, values,
711 replace_unknown_lazy=True)713 replace_unknown_lazy=True)
712714
713 self._add_to_alive(obj_info)715 self._add_to_alive(obj_info)
@@ -750,7 +752,8 @@
750 variable = obj_info.variables[column]752 variable = obj_info.variables[column]
751 lazy_value = variable.get_lazy()753 lazy_value = variable.get_lazy()
752 is_unknown_lazy = not (lazy_value is None or754 is_unknown_lazy = not (lazy_value is None or
753 lazy_value is AutoReload)755 lazy_value is AutoReload or
756 lazy_value is LazyGroupMember)
754 if keep_defined:757 if keep_defined:
755 if variable.is_defined() or is_unknown_lazy:758 if variable.is_defined() or is_unknown_lazy:
756 continue759 continue
@@ -871,6 +874,14 @@
871 the store, and then set all variables set to AutoReload to874 the store, and then set all variables set to AutoReload to
872 their database values.875 their database values.
873 """876 """
877 if lazy_value is LazyGroupMember:
878 fellow_columns = obj_info.cls_info.lazy_groups[variable.column]
879 for column in fellow_columns:
880 if obj_info.variables[column].get_lazy() is LazyGroupMember:
881 obj_info.variables[column].set(AutoReload)
882 # now that it's an AutoReload, resolve the lazy value
883 return self._resolve_lazy_value(obj_info, variable, AutoReload)
884
874 if lazy_value is not AutoReload and not isinstance(lazy_value, Expr):885 if lazy_value is not AutoReload and not isinstance(lazy_value, Expr):
875 # It's not something we handle.886 # It's not something we handle.
876 return887 return
@@ -1678,7 +1689,7 @@
1678 if isinstance(info, Column):1689 if isinstance(info, Column):
1679 default_tables.append(info.table)1690 default_tables.append(info.table)
1680 else:1691 else:
1681 columns.extend(info.columns)1692 columns.extend(info.eager_columns)
1682 default_tables.append(info.table)1693 default_tables.append(info.table)
1683 return columns, default_tables1694 return columns, default_tables
16841695
@@ -1706,7 +1717,7 @@
1706 value=values[values_start], from_db=True)1717 value=values[values_start], from_db=True)
1707 objects.append(variable.get())1718 objects.append(variable.get())
1708 else:1719 else:
1709 values_end += len(info.columns)1720 values_end += len(info.eager_columns)
1710 obj = store._load_object(info, result,1721 obj = store._load_object(info, result,
1711 values[values_start:values_end])1722 values[values_start:values_end])
1712 objects.append(obj)1723 objects.append(obj)
@@ -1809,3 +1820,21 @@
1809 pass1820 pass
18101821
1811AutoReload = AutoReload()1822AutoReload = AutoReload()
1823
1824
1825class LazyGroupMember(LazyValue):
1826 """A marker for indicating that this value is loaded on first access.
1827
1828 This is used internally when you make a property lazy, as in::
1829
1830 class Person(object):
1831 __storm_table__ = "person"
1832 id = Int(primary=True)
1833 name = Unicode(lazy=True)
1834
1835 When a C{Person} is retrieved from the store, it's C{name} property
1836 will be set to L{LazyGroupMember}.
1837 """
1838 pass
1839
1840LazyGroupMember = LazyGroupMember()
1812\ No newline at end of file1841\ No newline at end of file
18131842
=== modified file 'tests/info.py'
--- tests/info.py 2011-12-07 11:57:07 +0000
+++ tests/info.py 2011-12-29 16:23:23 +0000
@@ -22,7 +22,7 @@
22import gc22import gc
2323
24from storm.exceptions import ClassInfoError24from storm.exceptions import ClassInfoError
25from storm.properties import Property25from storm.properties import Property, Group
26from storm.variables import Variable26from storm.variables import Variable
27from storm.expr import Undef, Select, compile27from storm.expr import Undef, Select, compile
28from storm.info import *28from storm.info import *
@@ -168,6 +168,40 @@
168 cls_info = ClassInfo(Class)168 cls_info = ClassInfo(Class)
169 self.assertEquals(cls_info.primary_key_pos, (2, 0))169 self.assertEquals(cls_info.primary_key_pos, (2, 0))
170170
171 def test_eager_columns(self):
172 class Class(object):
173 __storm_table__ = 'table'
174 prop1 = Property(primary=True)
175 prop2 = Property()
176 prop3 = Property(lazy=True)
177 prop4 = Property(lazy=1)
178 prop5 = Property(lazy=Group())
179 cls_info = ClassInfo(Class)
180 self.assertEquals(cls_info.eager_columns, (Class.prop1, Class.prop2))
181
182 def test_lazy_columns(self):
183 class Class(object):
184 __storm_table__ = 'table'
185 prop1 = Property(primary=True)
186 prop2 = Property()
187 prop3 = Property(lazy=True)
188 prop4 = Property(lazy=1)
189 prop5 = Property(lazy=Group())
190 prop6 = Property(lazy=1)
191 cls_info = ClassInfo(Class)
192 self.assertEquals(cls_info.lazy_columns,
193 (Class.prop3, Class.prop4, Class.prop5, Class.prop6))
194 self.assertEqual(cls_info.lazy_groups[Class.prop3],
195 [Class.prop3])
196 self.assertEqual(cls_info.lazy_groups[Class.prop4],
197 [Class.prop4, Class.prop6])
198 self.assertEqual(cls_info.lazy_groups[Class.prop5],
199 [Class.prop5])
200 self.assertEqual(cls_info.lazy_groups[Class.prop6],
201 cls_info.lazy_groups[Class.prop4])
202
203
204
171205
172class ObjectInfoTest(TestHelper):206class ObjectInfoTest(TestHelper):
173207
174208
=== modified file 'tests/store/base.py'
--- tests/store/base.py 2011-08-01 12:54:36 +0000
+++ tests/store/base.py 2011-12-29 16:23:23 +0000
@@ -42,7 +42,8 @@
42 NoStoreError, NotFlushedError, NotOneError, OrderLoopError, UnorderedError,42 NoStoreError, NotFlushedError, NotOneError, OrderLoopError, UnorderedError,
43 WrongStoreError, DisconnectionError)43 WrongStoreError, DisconnectionError)
44from storm.cache import Cache44from storm.cache import Cache
45from storm.store import AutoReload, EmptyResultSet, Store, ResultSet45from storm.store import (
46 AutoReload, EmptyResultSet, Store, ResultSet, LazyGroupMember)
46from storm.tracer import debug47from storm.tracer import debug
4748
48from tests.info import Wrapper49from tests.info import Wrapper
@@ -130,6 +131,14 @@
130 id = Int(primary=True)131 id = Int(primary=True)
131 value = Decimal()132 value = Decimal()
132133
134class Sloth(object):
135 __storm_table__ = "sloth"
136 id = Int(primary=True)
137 val1 = Int()
138 val2 = Int(lazy=2)
139 val3 = Int(lazy=2)
140 val4 = Int(lazy=True)
141
133142
134class DecorateVariable(Variable):143class DecorateVariable(Variable):
135144
@@ -244,6 +253,8 @@
244 " VALUES (8, 20, 1, 4)")253 " VALUES (8, 20, 1, 4)")
245 connection.execute("INSERT INTO foovalue (id, foo_id, value1, value2)"254 connection.execute("INSERT INTO foovalue (id, foo_id, value1, value2)"
246 " VALUES (9, 20, 1, 2)")255 " VALUES (9, 20, 1, 2)")
256 connection.execute("INSERT INTO sloth (id, val1, val2, val3, val4)"
257 " VALUES (1, 1, 2, 3, 4)")
247258
248 connection.commit()259 connection.commit()
249260
@@ -5291,6 +5302,64 @@
5291 self.assertEquals(lazy_value, AutoReload)5302 self.assertEquals(lazy_value, AutoReload)
5292 self.assertEquals(foo.title, u"Default Title")5303 self.assertEquals(foo.title, u"Default Title")
52935304
5305 def test_lazy_loading_find(self):
5306 """
5307 Accessing a lazily-loaded attribute will cause it to load as well as
5308 any other attribute in its group.
5309 """
5310 sloth = self.store.find(Sloth, id=1).one()
5311 variables = get_obj_info(sloth).variables
5312 self.assertEqual(variables[Sloth.val2].get_lazy(), LazyGroupMember)
5313 self.assertEqual(variables[Sloth.val3].get_lazy(), LazyGroupMember)
5314 self.assertEqual(variables[Sloth.val4].get_lazy(), LazyGroupMember)
5315 val2 = sloth.val2
5316 # val2 and val3 are in a group together
5317 self.assertEqual(variables[Sloth.val2].get_lazy(), None)
5318 self.assertEqual(variables[Sloth.val3].get_lazy(), None)
5319 self.assertEqual(variables[Sloth.val4].get_lazy(), LazyGroupMember)
5320 self.assertEqual(val2, 2)
5321 self.assertEqual(sloth.val3, 3)
5322 val4 = sloth.val4
5323 self.assertEqual(variables[Sloth.val4].get_lazy(), None)
5324 self.assertEqual(val4, 4)
5325
5326 def test_lazy_loading_get(self):
5327 """
5328 get() should honor lazy attributes and not get them
5329 """
5330 sloth = self.store.get(Sloth, 1)
5331 variables = get_obj_info(sloth).variables
5332 self.assertEqual(variables[Sloth.val2].get_lazy(), LazyGroupMember)
5333 self.assertEqual(variables[Sloth.val3].get_lazy(), LazyGroupMember)
5334 self.assertEqual(variables[Sloth.val4].get_lazy(), LazyGroupMember)
5335 val2 = sloth.val2
5336 # val2 and val3 are in a group together
5337 self.assertEqual(variables[Sloth.val2].get_lazy(), None)
5338 self.assertEqual(variables[Sloth.val3].get_lazy(), None)
5339 self.assertEqual(variables[Sloth.val4].get_lazy(), LazyGroupMember)
5340 self.assertEqual(val2, 2)
5341 self.assertEqual(sloth.val3, 3)
5342 val4 = sloth.val4
5343 self.assertEqual(variables[Sloth.val4].get_lazy(), None)
5344 self.assertEqual(val4, 4)
5345
5346 def test_lazy_loading_set(self):
5347 """
5348 Setting an attribute should not cause a group to load, but should
5349 prevent that value from being overwritten by a later load.
5350 """
5351 sloth = self.store.find(Sloth, id=1).one()
5352 sloth.val2 = 200
5353 variables = get_obj_info(sloth).variables
5354 self.assertEqual(variables[Sloth.val2].get_lazy(), None)
5355 self.assertEqual(variables[Sloth.val3].get_lazy(), LazyGroupMember,
5356 "The other column in this lazy group should not yet "
5357 "be loaded.")
5358 val3 = sloth.val3
5359 self.assertEqual(val3, 3)
5360 self.assertEqual(sloth.val2, 200, "Attribute should not be overwritten"
5361 " by database value: %r" % sloth.val2)
5362
5294 def test_reference_break_on_local_diverged_doesnt_autoreload(self):5363 def test_reference_break_on_local_diverged_doesnt_autoreload(self):
5295 foo = self.store.get(Foo, 10)5364 foo = self.store.get(Foo, 10)
5296 self.store.autoreload(foo)5365 self.store.autoreload(foo)
52975366
=== modified file 'tests/store/mysql.py'
--- tests/store/mysql.py 2011-02-14 12:17:54 +0000
+++ tests/store/mysql.py 2011-12-29 16:23:23 +0000
@@ -79,6 +79,11 @@
79 connection.execute("CREATE TABLE unique_id "79 connection.execute("CREATE TABLE unique_id "
80 "(id VARCHAR(36) PRIMARY KEY) "80 "(id VARCHAR(36) PRIMARY KEY) "
81 "ENGINE=InnoDB")81 "ENGINE=InnoDB")
82 connection.execute("CREATE TABLE sloth "
83 "(id INT PRIMARY KEY AUTO_INCREMENT,"
84 " val1 INTEGER, val2 INTEGER, val3 INTEGER,"
85 " val4 INTEGER) "
86 "ENGINE=InnoDB")
82 connection.commit()87 connection.commit()
8388
8489
8590
=== modified file 'tests/store/postgres.py'
--- tests/store/postgres.py 2011-02-14 12:17:54 +0000
+++ tests/store/postgres.py 2011-12-29 16:23:23 +0000
@@ -93,6 +93,9 @@
93 " value1 INTEGER, value2 INTEGER)")93 " value1 INTEGER, value2 INTEGER)")
94 connection.execute("CREATE TABLE unique_id "94 connection.execute("CREATE TABLE unique_id "
95 "(id UUID PRIMARY KEY)")95 "(id UUID PRIMARY KEY)")
96 connection.execute("CREATE TABLE sloth "
97 "(id SERIAL PRIMARY KEY, val1 INTEGER,"
98 " val2 INTEGER, val3 INTEGER, val4 INTEGER)")
96 connection.commit()99 connection.commit()
97100
98 def drop_tables(self):101 def drop_tables(self):
99102
=== modified file 'tests/store/sqlite.py'
--- tests/store/sqlite.py 2011-02-14 12:17:54 +0000
+++ tests/store/sqlite.py 2011-12-29 16:23:23 +0000
@@ -65,6 +65,9 @@
65 " value1 INTEGER, value2 INTEGER)")65 " value1 INTEGER, value2 INTEGER)")
66 connection.execute("CREATE TABLE unique_id "66 connection.execute("CREATE TABLE unique_id "
67 "(id VARCHAR PRIMARY KEY)")67 "(id VARCHAR PRIMARY KEY)")
68 connection.execute("CREATE TABLE sloth "
69 "(id INTEGER PRIMARY KEY, val1 INTEGER,"
70 " val2 INTEGER, val3 INTEGER, val4 INTEGER)")
68 connection.commit()71 connection.commit()
6972
70 def drop_tables(self):73 def drop_tables(self):

Subscribers

People subscribed via source and target branches

to status/vote changes: