Storm

Merge lp:~cjwatson/storm/restore-mysql into lp:storm

  1. restore-mysql
  2. Merge into trunk
Proposed by Colin Watson
Status: Merged
Merged at revision: 568
Proposed branch: lp:~cjwatson/storm/restore-mysql
Merge into: lp:storm
Prerequisite: lp:~cjwatson/storm/py27-py35-workarounds
Diff against target: 995 lines (+708/-46)
13 files modified
.bzrignore (+1/-0)
MANIFEST.in (+1/-1)
NEWS (+1/-0)
README (+22/-41)
dev/test (+123/-2)
dev/ubuntu-deps (+3/-0)
setup.py (+2/-0)
storm/databases/mysql.py (+246/-0)
storm/docs/tutorial.rst (+2/-2)
storm/tests/databases/base.py (+2/-0)
storm/tests/databases/mysql.py (+167/-0)
storm/tests/store/base.py (+30/-0)
storm/tests/store/mysql.py (+108/-0)
To merge this branch: bzr merge lp:~cjwatson/storm/restore-mysql
Reviewer Review Type Date Requested Status
Ioana Lasc (community) Approve
Storm Developers Pending
Review via email: mp+400633@code.launchpad.net

Commit message

Restore MySQL support.

Description of the change

We've had some requests to put this back, and it's reasonably straightforward to support. The only tricky bit was arranging for automatic test database provisioning; the only near-equivalent of postgresfixture that I could find is pytest-specific, but it's not too hard to inline the necessary setup and teardown code.

MySQL support was dropped in https://code.launchpad.net/~ack/storm/drop-mysql-support/+merge/288652, but it's clear that the people doing that work were labouring under the assumption that Storm is only used within Canonical, which is demonstrably untrue.

To post a comment you must log in.
Revision history for this message
Colin Watson (cjwatson) wrote :

https://paste.ubuntu.com/p/CpVtdGRZgm/ is a diff of the three "new" (restored) files in this branch against the revision immediately before their previous removal, which may make them easier to review. As you can see, there are a few changes for Python 3 support, test directory reorganization, and to support automatic test database provisioning, but they're relatively short.

Revision history for this message
Ioana Lasc (ilasc) wrote :

LGTM

review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file '.bzrignore'
2--- .bzrignore 2020-05-26 10:28:24 +0000
3+++ .bzrignore 2021-04-06 09:31:39 +0000
4@@ -3,6 +3,7 @@
5 storm.egg-info
6 build-stamp
7 db
8+db-mysql
9 .db.lock
10 dist
11 debian/files
12
13=== modified file 'MANIFEST.in'
14--- MANIFEST.in 2020-05-26 10:28:24 +0000
15+++ MANIFEST.in 2021-04-06 09:31:39 +0000
16@@ -5,4 +5,4 @@
17 include storm/docs/Makefile
18 prune storm/docs/_build
19
20-prune db
21+prune db db-mysql
22
23=== modified file 'NEWS'
24--- NEWS 2020-12-07 22:38:36 +0000
25+++ NEWS 2021-04-06 09:31:39 +0000
26@@ -7,6 +7,7 @@
27 - Add optional case_sensitive argument to Comparable.startswith,
28 Comparable.endswith, and Comparable.contains_string. This is only
29 supported in the PostgreSQL backend.
30+- Restore MySQL support.
31
32 0.24 (2020-06-12)
33 =================
34
35=== modified file 'README'
36--- README 2020-05-26 10:28:24 +0000
37+++ README 2021-04-06 09:31:39 +0000
38@@ -22,7 +22,8 @@
39 * Storm is well designed (different classes have very clear
40 boundaries, with small and clean public APIs).
41 * Designed from day one to work both with thin relational
42- databases, such as SQLite, and big iron systems like PostgreSQL.
43+ databases, such as SQLite, and big iron systems like PostgreSQL
44+ and MySQL.
45 * Storm is easy to debug, since its code is written with a KISS
46 principle, and thus is easy to understand.
47 * Designed from day one to work both at the low end, with trivial
48@@ -89,6 +90,8 @@
49 is below.
50
51 $ dev/ubuntu-deps
52+ $ echo "$PWD/** rwk," | sudo tee /etc/apparmor.d/local/usr.sbin.mysqld >/dev/null
53+ $ sudo aa-enforce /usr/sbin/mysqld
54 $ make develop
55 $ make check
56
57@@ -103,15 +106,15 @@
58 The following instructions assume that you're using Ubuntu. The same procedure
59 will probably work without changes on a Debian system and with minimal changes
60 on a non-Debian-based linux distribution. In order to run the test suite, and
61-exercise all supported backends, you will need to install PostgreSQL, along
62-with the related Python database drivers:
63+exercise all supported backends, you will need to install MySQL and
64+PostgreSQL, along with the related Python database drivers:
65
66 $ sudo apt-get install \
67+ mysql-server \
68 postgresql pgbouncer \
69 build-essential
70
71-These will take a few minutes to download (its a bit under 200MB all
72-together).
73+These will take a few minutes to download.
74
75 The Python dependencies for running tests can mostly be installed with
76 apt-get:
77@@ -135,42 +138,20 @@
78 This ensures that all dependencies are available, downloading from
79 PyPI as appropriate.
80
81-Setting up database users and access security
82----------------------------------------------
83-
84-PostgreSQL needs to be setup to allow TCP/IP connections from
85-localhost. Edit /etc/postgresql/8.3/main/pg_hba.conf and make sure
86-the following line is present:
87-
88- host all all 127.0.0.1/32 trust
89-
90-This will probably (with PostgresSQL 8.4) entail changing 'md5' to
91-'trust'.
92-
93-In order to run the two-phase commit tests, you will also need to
94-change the max_prepared_transactions value in postgres.conf to
95-something like
96-
97- max_prepared_transactions = 200
98-
99-Now save and close, then restart the server:
100-
101- $ sudo /etc/init.d/postgresql-8.4 restart
102-
103-Lets create our PostgreSQL user now. As noted in the Ubuntu PostgreSQL
104-documentation, the easiest thing is to create a user with the same name as your
105-username. Run the following command to create a user for yourself (if prompted
106-for a password, leave it blank):
107-
108- $ sudo -u postgres createuser --superuser $USER
109-
110-Creating test databases
111------------------------
112-
113-The test suite needs some local databases in place to exercise PostgreSQL
114-functionality. Run:
115-
116- $ createdb storm_test
117+Database setup
118+--------------
119+
120+Most database setup is done automatically by the test suite. However,
121+Ubuntu's default MySQL packaging ships an AppArmor profile that prevents it
122+from writing to a local data directory. To allow the test suite to do this,
123+you will need to grant it access, which is most easily done by adding a line
124+such as this to /etc/apparmor.d/local/usr.sbin.mysqld:
125+
126+ /path/to/storm/** rwk,
127+
128+Then reload the profile:
129+
130+ $ sudo aa-enforce /usr/sbin/mysqld
131
132 Running the tests
133 -----------------
134
135=== modified file 'dev/test'
136--- dev/test 2019-11-21 01:57:01 +0000
137+++ dev/test 2021-04-06 09:31:39 +0000
138@@ -21,10 +21,16 @@
139 #
140 import glob
141 import optparse
142+import os
143 import re
144+import shutil
145+import socket
146+import subprocess
147+import sys
148+import time
149 import unittest
150-import sys
151-import os
152+
153+from pkg_resources import parse_version
154
155
156 def add_eggs_to_path():
157@@ -111,6 +117,120 @@
158 return wrapper
159
160
161+def with_mysql(runner_func):
162+ """If possible, wrap a test runner with code to set up MySQL.
163+
164+ Loosely based on the approach taken by pytest-mysql, although
165+ implemented separately.
166+ """
167+ try:
168+ import MySQLdb
169+ except ImportError:
170+ return runner_func
171+
172+ from six.moves.urllib.parse import (
173+ urlencode,
174+ urlunsplit,
175+ )
176+
177+ def wrapper():
178+ basedir = os.path.abspath("db-mysql")
179+ datadir = os.path.join(basedir, "data")
180+ unix_socket = os.path.join(basedir, "mysql.sock")
181+ logfile = os.path.join(basedir, "mysql.log")
182+ if os.path.exists(basedir):
183+ shutil.rmtree(basedir)
184+ os.makedirs(basedir)
185+
186+ mysqld_version_output = subprocess.check_output(
187+ ["mysqld", "--version"], universal_newlines=True).rstrip("\n")
188+ version = re.search(
189+ r"Ver ([\d.]+)", mysqld_version_output, flags=re.I).group(1)
190+ if ("MariaDB" not in mysqld_version_output and
191+ parse_version(version) >= parse_version("5.7.6")):
192+ subprocess.check_call([
193+ "mysqld",
194+ "--initialize-insecure",
195+ "--datadir=%s" % datadir,
196+ "--tmpdir=%s" % basedir,
197+ "--log-error=%s" % logfile,
198+ ])
199+ else:
200+ subprocess.check_call([
201+ "mysql_install_db",
202+ "--datadir=%s" % datadir,
203+ "--tmpdir=%s" % basedir,
204+ ])
205+ with open("/dev/null", "w") as devnull:
206+ server_proc = subprocess.Popen([
207+ "mysqld_safe",
208+ "--datadir=%s" % datadir,
209+ "--pid-file=%s" % os.path.join(basedir, "mysql.pid"),
210+ "--socket=%s" % unix_socket,
211+ "--skip-networking",
212+ "--log-error=%s" % logfile,
213+ "--tmpdir=%s" % basedir,
214+ "--skip-syslog",
215+ # We don't care about durability of test data. Try to
216+ # persuade MySQL to agree.
217+ "--innodb-doublewrite=0",
218+ "--innodb-flush-log-at-trx-commit=0",
219+ "--innodb-flush-method=O_DIRECT_NO_FSYNC",
220+ "--skip-innodb-file-per-table",
221+ "--sync-binlog=0",
222+ ], stdout=devnull)
223+
224+ try:
225+ start_time = time.time()
226+ while time.time() < start_time + 60:
227+ code = server_proc.poll()
228+ if code is not None and code != 0:
229+ raise Exception("mysqld_base exited %d" % code)
230+
231+ sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
232+ try:
233+ sock.connect(unix_socket)
234+ break
235+ except socket.error:
236+ pass # try again
237+ finally:
238+ sock.close()
239+
240+ time.sleep(0.1)
241+
242+ try:
243+ connection = MySQLdb.connect(
244+ user="root", unix_socket=unix_socket)
245+ try:
246+ cursor = connection.cursor()
247+ try:
248+ cursor.execute(
249+ "CREATE DATABASE storm_test CHARACTER SET utf8;")
250+ finally:
251+ cursor.close()
252+ finally:
253+ connection.close()
254+ uri = urlunsplit(
255+ ("mysql", "root@localhost", "/storm_test",
256+ urlencode({"unix_socket": unix_socket}), ""))
257+ os.environ["STORM_MYSQL_URI"] = uri
258+ os.environ["STORM_MYSQL_HOST_URI"] = uri
259+ return runner_func()
260+ finally:
261+ subprocess.check_call([
262+ "mysqladmin",
263+ "--socket=%s" % unix_socket,
264+ "--user=root",
265+ "shutdown",
266+ ])
267+ finally:
268+ if server_proc.poll() is None:
269+ server_proc.kill()
270+ server_proc.wait()
271+
272+ return wrapper
273+
274+
275 if __name__ == "__main__":
276 runner = os.environ.get("STORM_TEST_RUNNER")
277 if not runner:
278@@ -119,6 +239,7 @@
279 if not runner_func:
280 sys.exit("Test runner not found: %s" % runner)
281 runner_func = with_postgresfixture(runner_func)
282+ runner_func = with_mysql(runner_func)
283 sys.exit(runner_func())
284
285 # vim:ts=4:sw=4:et
286
287=== modified file 'dev/ubuntu-deps'
288--- dev/ubuntu-deps 2019-06-05 10:53:34 +0000
289+++ dev/ubuntu-deps 2021-04-06 09:31:39 +0000
290@@ -7,11 +7,14 @@
291
292 apt_get install --no-install-recommends \
293 build-essential \
294+ libmysqlclient-dev \
295 libpq-dev \
296+ mysql-server \
297 pgbouncer \
298 postgresql \
299 python-dev \
300 python-fixtures \
301+ python-mysqldb \
302 python-psycopg2 \
303 python-setuptools \
304 python-six \
305
306=== modified file 'setup.py'
307--- setup.py 2021-04-06 09:31:39 +0000
308+++ setup.py 2021-04-06 09:31:39 +0000
309@@ -25,6 +25,8 @@
310
311 tests_require = [
312 "fixtures >= 1.3.0",
313+ "mysqlclient",
314+ "mysqlclient < 2.0.0; python_version < '3'",
315 "pgbouncer >= 0.0.7",
316 "postgresfixture",
317 "psycopg2 >= 2.3.0",
318
319=== added file 'storm/databases/mysql.py'
320--- storm/databases/mysql.py 1970-01-01 00:00:00 +0000
321+++ storm/databases/mysql.py 2021-04-06 09:31:39 +0000
322@@ -0,0 +1,246 @@
323+#
324+# Copyright (c) 2006, 2007 Canonical
325+#
326+# Written by Gustavo Niemeyer <gustavo@niemeyer.net>
327+#
328+# This file is part of Storm Object Relational Mapper.
329+#
330+# Storm is free software; you can redistribute it and/or modify
331+# it under the terms of the GNU Lesser General Public License as
332+# published by the Free Software Foundation; either version 2.1 of
333+# the License, or (at your option) any later version.
334+#
335+# Storm is distributed in the hope that it will be useful,
336+# but WITHOUT ANY WARRANTY; without even the implied warranty of
337+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
338+# GNU Lesser General Public License for more details.
339+#
340+# You should have received a copy of the GNU Lesser General Public License
341+# along with this program. If not, see <http://www.gnu.org/licenses/>.
342+#
343+
344+from __future__ import print_function
345+
346+from datetime import time, timedelta
347+from array import array
348+import sys
349+
350+from storm.databases import dummy
351+
352+try:
353+ import MySQLdb
354+ import MySQLdb.converters
355+except ImportError:
356+ MySQLdb = dummy
357+
358+from storm.database import (
359+ Connection,
360+ ConnectionWrapper,
361+ Database,
362+ Result,
363+ )
364+from storm.exceptions import (
365+ DatabaseModuleError,
366+ OperationalError,
367+ wrap_exceptions,
368+ )
369+from storm.expr import (
370+ compile,
371+ compile_select,
372+ Insert,
373+ is_safe_token,
374+ Select,
375+ SQLToken,
376+ Undef,
377+ )
378+from storm.variables import Variable
379+
380+
381+compile = compile.create_child()
382+
383+@compile.when(Select)
384+def compile_select_mysql(compile, select, state):
385+ if select.offset is not Undef and select.limit is Undef:
386+ select.limit = sys.maxsize
387+ return compile_select(compile, select, state)
388+
389+@compile.when(SQLToken)
390+def compile_sql_token_mysql(compile, expr, state):
391+ """MySQL uses ` as the escape character by default."""
392+ if is_safe_token(expr) and not compile.is_reserved_word(expr):
393+ return expr
394+ return '`%s`' % expr.replace('`', '``')
395+
396+
397+class MySQLResult(Result):
398+
399+ @staticmethod
400+ def from_database(row):
401+ """Convert MySQL-specific datatypes to "normal" Python types.
402+
403+ If there are any C{array} instances in the row, convert them
404+ to strings.
405+ """
406+ for value in row:
407+ if isinstance(value, array):
408+ yield value.tostring()
409+ else:
410+ yield value
411+
412+
413+class MySQLConnection(Connection):
414+
415+ result_factory = MySQLResult
416+ param_mark = "%s"
417+ compile = compile
418+
419+ def execute(self, statement, params=None, noresult=False):
420+ if (isinstance(statement, Insert) and
421+ statement.primary_variables is not Undef):
422+
423+ result = Connection.execute(self, statement, params)
424+
425+ # The lastrowid value will be set if:
426+ # - the table had an AUTO INCREMENT column, and
427+ # - the column was not set during the insert or set to 0
428+ #
429+ # If these conditions are met, then lastrowid will be the
430+ # value of the first such column set. We assume that it
431+ # is the first undefined primary key variable.
432+ if result._raw_cursor.lastrowid:
433+ for variable in statement.primary_variables:
434+ if not variable.is_defined():
435+ variable.set(result._raw_cursor.lastrowid,
436+ from_db=True)
437+ break
438+ if noresult:
439+ result = None
440+ return result
441+ return Connection.execute(self, statement, params, noresult)
442+
443+ def to_database(self, params):
444+ for param in params:
445+ if isinstance(param, Variable):
446+ param = param.get(to_db=True)
447+ if isinstance(param, timedelta):
448+ yield str(param)
449+ else:
450+ yield param
451+
452+ def is_disconnection_error(self, exc, extra_disconnection_errors=()):
453+ # http://dev.mysql.com/doc/refman/5.0/en/gone-away.html
454+ return (isinstance(exc, (OperationalError,
455+ extra_disconnection_errors)) and
456+ exc.args[0] in (2006, 2013)) # (SERVER_GONE_ERROR, SERVER_LOST)
457+
458+
459+class MySQL(Database):
460+
461+ connection_factory = MySQLConnection
462+ _exception_module = MySQLdb
463+ _converters = None
464+
465+ def __init__(self, uri):
466+ super(MySQL, self).__init__(uri)
467+ if MySQLdb is dummy:
468+ raise DatabaseModuleError("'MySQLdb' module not found")
469+ self._connect_kwargs = {}
470+ if uri.database is not None:
471+ self._connect_kwargs["db"] = uri.database
472+ if uri.host is not None:
473+ self._connect_kwargs["host"] = uri.host
474+ if uri.port is not None:
475+ self._connect_kwargs["port"] = uri.port
476+ if uri.username is not None:
477+ self._connect_kwargs["user"] = uri.username
478+ if uri.password is not None:
479+ self._connect_kwargs["passwd"] = uri.password
480+ for option in ["unix_socket"]:
481+ if option in uri.options:
482+ self._connect_kwargs[option] = uri.options.get(option)
483+
484+ if self._converters is None:
485+ # MySQLdb returns a timedelta by default on TIME fields.
486+ converters = MySQLdb.converters.conversions.copy()
487+ converters[MySQLdb.converters.FIELD_TYPE.TIME] = _convert_time
488+ self.__class__._converters = converters
489+
490+ self._connect_kwargs["conv"] = self._converters
491+ self._connect_kwargs["use_unicode"] = True
492+ self._connect_kwargs["charset"] = uri.options.get("charset", "utf8")
493+
494+ def _raw_connect(self):
495+ raw_connection = ConnectionWrapper(
496+ MySQLdb.connect(**self._connect_kwargs), self)
497+
498+ # Here is another sad story about bad transactional behavior. MySQL
499+ # offers a feature to automatically reconnect dropped connections.
500+ # What sounds like a dream, is actually a nightmare for anyone who
501+ # is dealing with transactions. When a reconnection happens, the
502+ # currently running transaction is transparently rolled back, and
503+ # everything that was being done is lost, without notice. Not only
504+ # that, but the connection may be put back in AUTOCOMMIT mode, even
505+ # when that's not the default MySQLdb behavior. The MySQL developers
506+ # quickly understood that this is a terrible idea, and removed the
507+ # behavior in MySQL 5.0.3. Unfortunately, Debian and Ubuntu still
508+ # have a patch for the MySQLdb module which *reenables* that
509+ # behavior by default even past version 5.0.3 of MySQL.
510+ #
511+ # Some links:
512+ # http://dev.mysql.com/doc/refman/5.0/en/auto-reconnect.html
513+ # http://dev.mysql.com/doc/refman/5.0/en/mysql-reconnect.html
514+ # http://dev.mysql.com/doc/refman/5.0/en/gone-away.html
515+ #
516+ # What we do here is to explore something that is a very weird
517+ # side-effect, discovered by reading the code. When we call the
518+ # ping() with a False argument, the automatic reconnection is
519+ # disabled in a *permanent* way for this connection. The argument
520+ # to ping() is new in 1.2.2, though.
521+ if MySQLdb.version_info >= (1, 2, 2):
522+ raw_connection.ping(False)
523+
524+ return raw_connection
525+
526+ def raw_connect(self):
527+ with wrap_exceptions(self):
528+ return self._raw_connect()
529+
530+
531+create_from_uri = MySQL
532+
533+
534+def _convert_time(time_str):
535+ h, m, s = time_str.split(":")
536+ if "." in s:
537+ f = float(s)
538+ s = int(f)
539+ return time(int(h), int(m), s, (f-s)*1000000)
540+ return time(int(h), int(m), int(s), 0)
541+
542+
543+# --------------------------------------------------------------------
544+# Reserved words, MySQL specific
545+
546+# The list of reserved words here are MySQL specific. SQL92 reserved words
547+# are registered in storm.expr, near the "Reserved words, from SQL1992"
548+# comment. The reserved words here were taken from:
549+#
550+# http://dev.mysql.com/doc/refman/5.4/en/reserved-words.html
551+compile.add_reserved_words("""
552+ accessible analyze asensitive before bigint binary blob call change
553+ condition current_user database databases day_hour day_microsecond
554+ day_minute day_second delayed deterministic distinctrow div dual each
555+ elseif enclosed escaped exit explain float4 float8 force fulltext
556+ high_priority hour_microsecond hour_minute hour_second if ignore index
557+ infile inout int1 int2 int3 int4 int8 iterate keys kill leave limit linear
558+ lines load localtime localtimestamp lock long longblob longtext loop
559+ low_priority master_ssl_verify_server_cert mediumblob mediumint mediumtext
560+ middleint minute_microsecond minute_second mod modifies no_write_to_binlog
561+ optimize optionally out outfile purge range read_write reads regexp
562+ release rename repeat replace require return rlike schemas
563+ second_microsecond sensitive separator show spatial specific
564+ sql_big_result sql_calc_found_rows sql_small_result sqlexception
565+ sqlwarning ssl starting straight_join terminated tinyblob tinyint tinytext
566+ trigger undo unlock unsigned use utc_date utc_time utc_timestamp varbinary
567+ varcharacter while xor year_month zerofill
568+ """.split())
569
570=== modified file 'storm/docs/tutorial.rst'
571--- storm/docs/tutorial.rst 2020-05-26 10:28:24 +0000
572+++ storm/docs/tutorial.rst 2021-04-06 09:31:39 +0000
573@@ -36,7 +36,7 @@
574 >>> database = create_database("sqlite:")
575 >>> store = Store(database)
576
577-Two databases are supported at the moment: SQLite and PostgreSQL.
578+Three databases are supported at the moment: SQLite, MySQL, and PostgreSQL.
579 The parameter passed to :py:func:`~storm.database.create_database` is an
580 URI, as follows:
581
582@@ -45,7 +45,7 @@
583 # database = create_database(
584 # "scheme://username:password@hostname:port/database_name")
585
586-The ``scheme`` may be ``sqlite`` or ``postgres``.
587+The ``scheme`` may be ``sqlite``, ``mysql``, or ``postgres``.
588
589 Now we have to create the table that will actually hold the data
590 for our class.
591
592=== modified file 'storm/tests/databases/base.py'
593--- storm/tests/databases/base.py 2020-02-10 15:30:23 +0000
594+++ storm/tests/databases/base.py 2021-04-06 09:31:39 +0000
595@@ -333,6 +333,8 @@
596 that works with data in memory, in fact) becomes a dangerous thing.
597
598 For PostgreSQL, isolation level must be SERIALIZABLE.
599+ For MySQL, isolation level must be REPEATABLE READ (the default),
600+ and the InnoDB engine must be in use.
601 For SQLite, the isolation level already is SERIALIZABLE when not
602 in autocommit mode. OTOH, PySQLite is nuts regarding transactional
603 behavior, and will easily offer READ COMMITTED behavior inside a
604
605=== added file 'storm/tests/databases/mysql.py'
606--- storm/tests/databases/mysql.py 1970-01-01 00:00:00 +0000
607+++ storm/tests/databases/mysql.py 2021-04-06 09:31:39 +0000
608@@ -0,0 +1,167 @@
609+#
610+# Copyright (c) 2006, 2007 Canonical
611+#
612+# Written by Gustavo Niemeyer <gustavo@niemeyer.net>
613+#
614+# This file is part of Storm Object Relational Mapper.
615+#
616+# Storm is free software; you can redistribute it and/or modify
617+# it under the terms of the GNU Lesser General Public License as
618+# published by the Free Software Foundation; either version 2.1 of
619+# the License, or (at your option) any later version.
620+#
621+# Storm is distributed in the hope that it will be useful,
622+# but WITHOUT ANY WARRANTY; without even the implied warranty of
623+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
624+# GNU Lesser General Public License for more details.
625+#
626+# You should have received a copy of the GNU Lesser General Public License
627+# along with this program. If not, see <http://www.gnu.org/licenses/>.
628+#
629+from __future__ import print_function
630+
631+import os
632+
633+from six.moves.urllib.parse import urlunsplit
634+
635+from storm.databases.mysql import MySQL
636+from storm.database import create_database
637+from storm.expr import Column, Insert
638+from storm.uri import URI
639+from storm.variables import IntVariable, UnicodeVariable
640+
641+from storm.tests.databases.base import (
642+ DatabaseTest, DatabaseDisconnectionTest, UnsupportedDatabaseTest)
643+from storm.tests.databases.proxy import ProxyTCPServer
644+from storm.tests.helper import TestHelper
645+
646+
647+def create_proxy_and_uri(uri):
648+ """Create a TCP proxy to a Unix-domain database identified by `uri`."""
649+ proxy = ProxyTCPServer(uri.options["unix_socket"])
650+ proxy_host, proxy_port = proxy.server_address
651+ proxy_uri = URI(urlunsplit(
652+ ("mysql", "%s:%s" % (proxy_host, proxy_port), "/storm_test",
653+ "", "")))
654+ return proxy, proxy_uri
655+
656+
657+class MySQLTest(DatabaseTest, TestHelper):
658+
659+ supports_microseconds = False
660+
661+ def is_supported(self):
662+ return bool(os.environ.get("STORM_MYSQL_URI"))
663+
664+ def create_database(self):
665+ self.database = create_database(os.environ["STORM_MYSQL_URI"])
666+
667+ def create_tables(self):
668+ self.connection.execute("CREATE TABLE number "
669+ "(one INTEGER, two INTEGER, three INTEGER)")
670+ self.connection.execute("CREATE TABLE test "
671+ "(id INT AUTO_INCREMENT PRIMARY KEY,"
672+ " title VARCHAR(50)) ENGINE=InnoDB")
673+ self.connection.execute("CREATE TABLE datetime_test "
674+ "(id INT AUTO_INCREMENT PRIMARY KEY,"
675+ " dt TIMESTAMP, d DATE, t TIME, td TEXT) "
676+ "ENGINE=InnoDB")
677+ self.connection.execute("CREATE TABLE bin_test "
678+ "(id INT AUTO_INCREMENT PRIMARY KEY,"
679+ " b BLOB) ENGINE=InnoDB")
680+
681+ def test_wb_create_database(self):
682+ database = create_database("mysql://un:pw@ht:12/db?unix_socket=us")
683+ self.assertTrue(isinstance(database, MySQL))
684+ for key, value in [("db", "db"), ("host", "ht"), ("port", 12),
685+ ("user", "un"), ("passwd", "pw"),
686+ ("unix_socket", "us")]:
687+ self.assertEqual(database._connect_kwargs.get(key), value)
688+
689+ def test_charset_defaults_to_utf8(self):
690+ result = self.connection.execute("SELECT @@character_set_client")
691+ self.assertEqual(result.get_one(), ("utf8",))
692+
693+ def test_charset_option(self):
694+ uri = URI(os.environ["STORM_MYSQL_URI"])
695+ uri.options["charset"] = "ascii"
696+ database = create_database(uri)
697+ connection = database.connect()
698+ result = connection.execute("SELECT @@character_set_client")
699+ self.assertEqual(result.get_one(), ("ascii",))
700+
701+ def test_get_insert_identity(self):
702+ # Primary keys are filled in during execute() for MySQL
703+ pass
704+
705+ def test_get_insert_identity_composed(self):
706+ # Primary keys are filled in during execute() for MySQL
707+ pass
708+
709+ def test_execute_insert_auto_increment_primary_key(self):
710+ id_column = Column("id", "test")
711+ id_variable = IntVariable()
712+ title_column = Column("title", "test")
713+ title_variable = UnicodeVariable(u"testing")
714+
715+ # This is not part of the table. It is just used to show that
716+ # only one primary key variable is set from the insert ID.
717+ dummy_column = Column("dummy", "test")
718+ dummy_variable = IntVariable()
719+
720+ insert = Insert({title_column: title_variable},
721+ primary_columns=(id_column, dummy_column),
722+ primary_variables=(id_variable, dummy_variable))
723+ self.connection.execute(insert)
724+ self.assertTrue(id_variable.is_defined())
725+ self.assertFalse(dummy_variable.is_defined())
726+
727+ # The newly inserted row should have the maximum id value for
728+ # the table.
729+ result = self.connection.execute("SELECT MAX(id) FROM test")
730+ self.assertEqual(result.get_one()[0], id_variable.get())
731+
732+ def test_mysql_specific_reserved_words(self):
733+ reserved_words = """
734+ accessible analyze asensitive before bigint binary blob call
735+ change condition current_user database databases day_hour
736+ day_microsecond day_minute day_second delayed deterministic
737+ distinctrow div dual each elseif enclosed escaped exit explain
738+ float4 float8 force fulltext high_priority hour_microsecond
739+ hour_minute hour_second if ignore index infile inout int1 int2
740+ int3 int4 int8 iterate keys kill leave limit linear lines load
741+ localtime localtimestamp lock long longblob longtext loop
742+ low_priority master_ssl_verify_server_cert mediumblob mediumint
743+ mediumtext middleint minute_microsecond minute_second mod modifies
744+ no_write_to_binlog optimize optionally out outfile purge range
745+ read_write reads regexp release rename repeat replace require
746+ return rlike schemas second_microsecond sensitive separator show
747+ spatial specific sql_big_result sql_calc_found_rows
748+ sql_small_result sqlexception sqlwarning ssl starting
749+ straight_join terminated tinyblob tinyint tinytext trigger undo
750+ unlock unsigned use utc_date utc_time utc_timestamp varbinary
751+ varcharacter while xor year_month zerofill
752+ """.split()
753+ for word in reserved_words:
754+ self.assertTrue(self.connection.compile.is_reserved_word(word),
755+ "Word missing: %s" % (word,))
756+
757+
758+class MySQLUnsupportedTest(UnsupportedDatabaseTest, TestHelper):
759+
760+ dbapi_module_names = ["MySQLdb"]
761+ db_module_name = "mysql"
762+
763+
764+class MySQLDisconnectionTest(DatabaseDisconnectionTest, TestHelper):
765+
766+ environment_variable = "STORM_MYSQL_URI"
767+ host_environment_variable = "STORM_MYSQL_HOST_URI"
768+ default_port = 3306
769+
770+ def create_proxy(self, uri):
771+ """See `DatabaseDisconnectionMixin.create_proxy`."""
772+ if "unix_socket" in uri.options:
773+ return create_proxy_and_uri(uri)[0]
774+ else:
775+ return super(MySQLDisconnectionTest, self).create_proxy(uri)
776
777=== modified file 'storm/tests/store/base.py'
778--- storm/tests/store/base.py 2020-03-18 16:50:12 +0000
779+++ storm/tests/store/base.py 2021-04-06 09:31:39 +0000
780@@ -901,6 +901,8 @@
781 result2 = self.store.find(Foo, Foo.id != 10)
782 self.assertEqual(foo in result1.union(result2), True)
783
784+ if self.__class__.__name__.startswith("MySQL"):
785+ return
786 self.assertEqual(foo in result1.intersection(result2), False)
787 self.assertEqual(foo in result1.intersection(result1), True)
788 self.assertEqual(foo in result1.difference(result2), True)
789@@ -1352,6 +1354,8 @@
790 result2 = self.store.find((Foo, Bar), Bar.foo_id == Foo.id)
791 self.assertEqual((foo, bar) in result1.union(result2), True)
792
793+ if self.__class__.__name__.startswith("MySQL"):
794+ return
795 self.assertEqual((foo, bar) in result1.intersection(result2), True)
796 self.assertEqual((foo, bar) in result1.difference(result2), False)
797
798@@ -1468,6 +1472,8 @@
799 result2 = self.store.find(Foo.title)
800 self.assertEqual(u"Title 10" in result1.union(result2), True)
801
802+ if self.__class__.__name__.startswith("MySQL"):
803+ return
804 self.assertEqual(u"Title 10" in result1.intersection(result2), True)
805 self.assertEqual(u"Title 10" in result1.difference(result2), False)
806
807@@ -5733,6 +5739,9 @@
808 self.assertEqual(result3.avg(Foo.id), 10)
809
810 def test_result_difference(self):
811+ if self.__class__.__name__.startswith("MySQL"):
812+ self.skipTest("Skipping ResultSet.difference tests on MySQL")
813+
814 result1 = self.store.find(Foo)
815 result2 = self.store.find(Foo, id=20)
816 result3 = result1.difference(result2)
817@@ -5750,6 +5759,9 @@
818 ])
819
820 def test_result_difference_with_empty(self):
821+ if self.__class__.__name__.startswith("MySQL"):
822+ self.skipTest("Skipping ResultSet.difference tests on MySQL")
823+
824 result1 = self.store.find(Foo, id=30)
825 result2 = EmptyResultSet()
826
827@@ -5760,11 +5772,17 @@
828 ])
829
830 def test_result_difference_incompatible(self):
831+ if self.__class__.__name__.startswith("MySQL"):
832+ self.skipTest("Skipping ResultSet.difference tests on MySQL")
833+
834 result1 = self.store.find(Foo, id=10)
835 result2 = self.store.find(Bar, id=100)
836 self.assertRaises(FeatureError, result1.difference, result2)
837
838 def test_result_difference_count(self):
839+ if self.__class__.__name__.startswith("MySQL"):
840+ self.skipTest("Skipping ResultSet.difference tests on MySQL")
841+
842 result1 = self.store.find(Foo)
843 result2 = self.store.find(Foo, id=20)
844
845@@ -5782,6 +5800,9 @@
846 self.assertEqual(result2.count(), 3)
847
848 def test_result_intersection(self):
849+ if self.__class__.__name__.startswith("MySQL"):
850+ self.skipTest("Skipping ResultSet.intersection tests on MySQL")
851+
852 result1 = self.store.find(Foo)
853 result2 = self.store.find(Foo, Foo.id.is_in((10, 30)))
854 result3 = result1.intersection(result2)
855@@ -5799,6 +5820,9 @@
856 ])
857
858 def test_result_intersection_with_empty(self):
859+ if self.__class__.__name__.startswith("MySQL"):
860+ self.skipTest("Skipping ResultSet.intersection tests on MySQL")
861+
862 result1 = self.store.find(Foo, id=30)
863 result2 = EmptyResultSet()
864 result3 = result1.intersection(result2)
865@@ -5806,11 +5830,17 @@
866 self.assertEqual(len(list(result3)), 0)
867
868 def test_result_intersection_incompatible(self):
869+ if self.__class__.__name__.startswith("MySQL"):
870+ self.skipTest("Skipping ResultSet.intersection tests on MySQL")
871+
872 result1 = self.store.find(Foo, id=10)
873 result2 = self.store.find(Bar, id=100)
874 self.assertRaises(FeatureError, result1.intersection, result2)
875
876 def test_result_intersection_count(self):
877+ if self.__class__.__name__.startswith("MySQL"):
878+ self.skipTest("Skipping ResultSet.intersection tests on MySQL")
879+
880 result1 = self.store.find(Foo, Foo.id.is_in((10, 20)))
881 result2 = self.store.find(Foo, Foo.id.is_in((10, 30)))
882 result3 = result1.intersection(result2)
883
884=== added file 'storm/tests/store/mysql.py'
885--- storm/tests/store/mysql.py 1970-01-01 00:00:00 +0000
886+++ storm/tests/store/mysql.py 2021-04-06 09:31:39 +0000
887@@ -0,0 +1,108 @@
888+#
889+# Copyright (c) 2006, 2007 Canonical
890+#
891+# Written by Gustavo Niemeyer <gustavo@niemeyer.net>
892+#
893+# This file is part of Storm Object Relational Mapper.
894+#
895+# Storm is free software; you can redistribute it and/or modify
896+# it under the terms of the GNU Lesser General Public License as
897+# published by the Free Software Foundation; either version 2.1 of
898+# the License, or (at your option) any later version.
899+#
900+# Storm is distributed in the hope that it will be useful,
901+# but WITHOUT ANY WARRANTY; without even the implied warranty of
902+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
903+# GNU Lesser General Public License for more details.
904+#
905+# You should have received a copy of the GNU Lesser General Public License
906+# along with this program. If not, see <http://www.gnu.org/licenses/>.
907+#
908+from __future__ import print_function
909+
910+import os
911+
912+from storm.database import create_database
913+
914+from storm.tests.store.base import StoreTest, EmptyResultSetTest
915+from storm.tests.helper import TestHelper
916+
917+
918+class MySQLStoreTest(TestHelper, StoreTest):
919+
920+ def setUp(self):
921+ TestHelper.setUp(self)
922+ StoreTest.setUp(self)
923+
924+ def tearDown(self):
925+ TestHelper.tearDown(self)
926+ StoreTest.tearDown(self)
927+
928+ def is_supported(self):
929+ return bool(os.environ.get("STORM_MYSQL_URI"))
930+
931+ def create_database(self):
932+ self.database = create_database(os.environ["STORM_MYSQL_URI"])
933+
934+ def create_tables(self):
935+ connection = self.connection
936+ connection.execute("CREATE TABLE foo "
937+ "(id INT PRIMARY KEY AUTO_INCREMENT,"
938+ " title VARCHAR(50) DEFAULT 'Default Title') "
939+ "ENGINE=InnoDB")
940+ connection.execute("CREATE TABLE bar "
941+ "(id INT PRIMARY KEY AUTO_INCREMENT,"
942+ " foo_id INTEGER, title VARCHAR(50)) "
943+ "ENGINE=InnoDB")
944+ connection.execute("CREATE TABLE bin "
945+ "(id INT PRIMARY KEY AUTO_INCREMENT,"
946+ " bin BLOB, foo_id INTEGER) "
947+ "ENGINE=InnoDB")
948+ connection.execute("CREATE TABLE link "
949+ "(foo_id INTEGER, bar_id INTEGER,"
950+ " PRIMARY KEY (foo_id, bar_id)) "
951+ "ENGINE=InnoDB")
952+ connection.execute("CREATE TABLE money "
953+ "(id INT PRIMARY KEY AUTO_INCREMENT,"
954+ " value NUMERIC(6,4)) "
955+ "ENGINE=InnoDB")
956+ connection.execute("CREATE TABLE selfref "
957+ "(id INT PRIMARY KEY AUTO_INCREMENT,"
958+ " title VARCHAR(50),"
959+ " selfref_id INTEGER,"
960+ " INDEX (selfref_id),"
961+ " FOREIGN KEY (selfref_id) REFERENCES selfref(id)) "
962+ "ENGINE=InnoDB")
963+ connection.execute("CREATE TABLE foovalue "
964+ "(id INT PRIMARY KEY AUTO_INCREMENT,"
965+ " foo_id INTEGER,"
966+ " value1 INTEGER, value2 INTEGER) "
967+ "ENGINE=InnoDB")
968+ connection.execute("CREATE TABLE unique_id "
969+ "(id VARCHAR(36) PRIMARY KEY) "
970+ "ENGINE=InnoDB")
971+ connection.commit()
972+
973+
974+class MySQLEmptyResultSetTest(TestHelper, EmptyResultSetTest):
975+
976+ def setUp(self):
977+ TestHelper.setUp(self)
978+ EmptyResultSetTest.setUp(self)
979+
980+ def tearDown(self):
981+ TestHelper.tearDown(self)
982+ EmptyResultSetTest.tearDown(self)
983+
984+ def is_supported(self):
985+ return bool(os.environ.get("STORM_MYSQL_URI"))
986+
987+ def create_database(self):
988+ self.database = create_database(os.environ["STORM_MYSQL_URI"])
989+
990+ def create_tables(self):
991+ self.connection.execute("CREATE TABLE foo "
992+ "(id INT PRIMARY KEY AUTO_INCREMENT,"
993+ " title VARCHAR(50) DEFAULT 'Default Title') "
994+ "ENGINE=InnoDB")
995+ self.connection.commit()

Subscribers

People subscribed via source and target branches

to status/vote changes: