Storm

Merge lp:~ack/storm/postgres-json-support into lp:storm

  1. postgres-json-support
  2. Merge into trunk
Proposed by Alberto Donato
Status: Merged
Merged at revision: 482
Proposed branch: lp:~ack/storm/postgres-json-support
Merge into: lp:storm
Diff against target: 109 lines (+56/-3)
2 files modified
storm/databases/postgres.py (+27/-1)
tests/databases/postgres.py (+29/-2)
To merge this branch: bzr merge lp:~ack/storm/postgres-json-support
Reviewer Review Type Date Requested Status
Stuart Bishop (community) Approve
Free Ekanayaka (community) Approve
Review via email: mp+293255@code.launchpad.net

Description of the change

This adds a Postgres-specific version of the JSON property (and JSONVariable).
It's supposed to be used with the Postgres "json" type.

Note that psycopg2 >= 2.5 automatically decodes json values to python objects, while previous versions return a plain string.

To post a comment you must log in.
lp:~ack/storm/postgres-json-support updated
483. By Alberto Donato

Lint

Revision history for this message
Stuart Bishop (stub) wrote :

This looks good, but seems to be only half of the picture. You can pull JSON out of the DB and it is correctly deserialized into the relevant Python object. However, you can't store a Python object and have it automatically serialized to JSON - you need to do that yourself (and by choosing this approach, it can't really be added in later).

Can we also add a _dumps() method (or whatever it needs to be called) to serialize the value to JSON (or raise an exception if it is unserializable)?

review: Needs Information
Revision history for this message
Free Ekanayaka (free.ekanayaka) wrote :

+1 with a nit

review: Approve
lp:~ack/storm/postgres-json-support updated
484. By Alberto Donato

Address Free's review.

Revision history for this message
Alberto Donato (ack) wrote :

@Stuart, the postgres-specific JSON property inherits from the generic one, which does the serialization in _dumps.
That works fine for both text and json columns, so it's not overridden in the pg one.

Revision history for this message
Stuart Bishop (stub) :
review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'storm/databases/postgres.py'
2--- storm/databases/postgres.py 2016-03-30 10:56:35 +0000
3+++ storm/databases/postgres.py 2016-04-29 08:38:59 +0000
4@@ -18,7 +18,10 @@
5 # You should have received a copy of the GNU Lesser General Public License
6 # along with this program. If not, see <http://www.gnu.org/licenses/>.
7 #
8+
9 from datetime import datetime, date, time, timedelta
10+import json
11+
12 from distutils.version import LooseVersion
13
14 from storm.databases import dummy
15@@ -43,7 +46,9 @@
16 Sequence, Like, SQLToken, BinaryOper, COLUMN, COLUMN_NAME, COLUMN_PREFIX,
17 TABLE, compile, compile_select, compile_insert, compile_set_expr,
18 compile_like, compile_sql_token)
19-from storm.variables import Variable, ListVariable
20+from storm.variables import (
21+ Variable, ListVariable, JSONVariable as BaseJSONVariable)
22+from storm.properties import SimpleProperty
23 from storm.database import Database, Connection, Result
24 from storm.exceptions import (
25 install_exceptions, DatabaseError, DatabaseModuleError, InterfaceError,
26@@ -470,3 +475,24 @@
27 """Return an element of a JSON value (by index or field name) as text."""
28
29 oper = "->>"
30+
31+
32+# Postgres-specific properties and variables
33+
34+class JSONVariable(BaseJSONVariable):
35+
36+ __slots__ = ()
37+
38+ def _loads(self, value):
39+ if isinstance(value, str):
40+ # psycopg versions < 2.5 don't automatically convert JSON columns
41+ # to python objects, they return a string.
42+ #
43+ # Note that on newer versions, if the object contained is an actual
44+ # string, it's returned as unicode, so the check is still valid.
45+ return json.loads(value)
46+ return value
47+
48+
49+class JSON(SimpleProperty):
50+ variable_class = JSONVariable
51
52=== modified file 'tests/databases/postgres.py'
53--- tests/databases/postgres.py 2016-03-30 15:22:11 +0000
54+++ tests/databases/postgres.py 2016-04-29 08:38:59 +0000
55@@ -24,8 +24,9 @@
56
57 from storm.databases.postgres import (
58 Postgres, compile, currval, Returning, Case, PostgresTimeoutTracer,
59- make_dsn, JSONElement, JSONTextElement)
60+ make_dsn, JSONElement, JSONTextElement, JSON)
61 from storm.database import create_database
62+from storm.store import Store
63 from storm.exceptions import InterfaceError, ProgrammingError
64 from storm.variables import DateTimeVariable, RawStrVariable
65 from storm.variables import ListVariable, IntVariable, Variable
66@@ -98,10 +99,14 @@
67 self.connection.execute("CREATE TABLE returning_test "
68 "(id1 INTEGER DEFAULT 123, "
69 " id2 INTEGER DEFAULT 456)")
70+ self.connection.execute("CREATE TABLE json_test "
71+ "(id SERIAL PRIMARY KEY, "
72+ " json JSON)")
73
74 def drop_tables(self):
75 super(PostgresTest, self).drop_tables()
76- for table in ["like_case_insensitive_test", "returning_test"]:
77+ tables = ("like_case_insensitive_test", "returning_test", "json_test")
78+ for table in tables:
79 try:
80 self.connection.execute("DROP TABLE %s" % table)
81 self.connection.commit()
82@@ -661,6 +666,28 @@
83 result = connection.execute(Select(Func("pg_typeof", expr)))
84 self.assertEqual("text", result.get_one()[0])
85
86+ def test_json_property(self):
87+ """The JSON property is encoded as JSON"""
88+
89+ class TestModel(object):
90+ __storm_table__ = "json_test"
91+
92+ id = Int(primary=True)
93+ json = JSON()
94+
95+ connection = self.database.connect()
96+ value = {"a": 3, "b": "foo", "c": None}
97+ db_value = json.dumps(value).decode("utf-8")
98+ connection.execute(
99+ "INSERT INTO json_test (json) VALUES (?)", (db_value,))
100+ connection.commit()
101+
102+ store = Store(self.database)
103+ obj = store.find(TestModel).one()
104+ store.close()
105+ # The JSON object is decoded to python
106+ self.assertEqual(value, obj.json)
107+
108
109 _max_prepared_transactions = None
110

Subscribers

People subscribed via source and target branches

to status/vote changes: