Drizzle

Merge lp:~drizzle-developers/drizzle/replication-docs-7.1 into lp:~drizzle-trunk/drizzle/development

  1. replication-docs-7.1
  2. Merge into development
Proposed by Brian Aker
Status: Merged
Approved by: Brian Aker
Approved revision: 2533
Merged at revision: 2531
Proposed branch: lp:~drizzle-developers/drizzle/replication-docs-7.1
Merge into: lp:~drizzle-trunk/drizzle/development
Diff against target: 3638 lines (+2031/-1287) (has conflicts)
37 files modified
docs/administration/authentication.rst (+2/-0)
docs/administration/authorization.rst (+2/-0)
docs/administration/index.rst (+4/-2)
docs/administration/plugins.rst (+3/-1)
docs/clients/drizzledump.rst (+2/-0)
docs/configuration/drizzled.rst (+37/-28)
docs/configuration/options.rst (+2/-2)
docs/functions/datetime.rst (+4/-0)
docs/index.rst (+7/-2)
docs/replication/appliers/index.rst (+33/-0)
docs/replication/appliers/rabbitmq.rst (+199/-0)
docs/replication/appliers/rabbitmq/changelog.rst (+13/-0)
docs/replication/appliers/slave.rst (+64/-0)
docs/replication/appliers/slave/administration.rst (+332/-0)
docs/replication/appliers/slave/changelog.rst (+27/-0)
docs/replication/appliers/slave/configuration.rst (+229/-0)
docs/replication/appliers/slave/details.rst (+11/-0)
docs/replication/appliers/zeromq.rst (+55/-0)
docs/replication/drizzle.rst (+0/-324)
docs/replication/examples/index.rst (+32/-0)
docs/replication/examples/rabbitmq/basic.rst (+55/-0)
docs/replication/examples/slave/provision_new_slave.rst (+61/-0)
docs/replication/examples/slave/simple_master_slave.rst (+172/-0)
docs/replication/index.rst (+172/-0)
docs/replication/messages/index.rst (+251/-0)
docs/replication/replicators/default.rst (+64/-0)
docs/replication/replicators/filtered.rst (+143/-0)
docs/replication/replicators/index.rst (+28/-0)
plugin/default_replicator/docs/index.rst (+1/-63)
plugin/filtered_replicator/docs/index.rst (+1/-142)
plugin/innobase/docs/index.rst (+19/-2)
plugin/rabbitmq/docs/index.rst (+3/-240)
plugin/slave/docs/admin.rst (+0/-136)
plugin/slave/docs/index.rst (+1/-162)
plugin/slave/docs/user_example.rst (+0/-134)
plugin/slave/module.cc (+1/-1)
plugin/zeromq/docs/index.rst (+1/-48)
Text conflict in docs/functions/datetime.rst
To merge this branch: bzr merge lp:~drizzle-developers/drizzle/replication-docs-7.1
Reviewer Review Type Date Requested Status
Drizzle Merge Team Pending
Review via email: mp+98078@code.launchpad.net
To post a comment you must log in.
Revision history for this message
Sekhar (msekharmca) wrote :

Please elaborate the content..

Thanks,
Sekhar

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== modified file 'docs/administration/authentication.rst'
--- docs/administration/authentication.rst 2011-11-14 22:29:19 +0000
+++ docs/administration/authentication.rst 2012-03-18 00:36:19 +0000
@@ -1,3 +1,5 @@
1.. _authentication:
2
1Authentication3Authentication
2==============4==============
35
46
=== modified file 'docs/administration/authorization.rst'
--- docs/administration/authorization.rst 2011-11-14 22:29:19 +0000
+++ docs/administration/authorization.rst 2012-03-18 00:36:19 +0000
@@ -1,3 +1,5 @@
1.. _authorization:
2
1Authorization3Authorization
2=============4=============
35
46
=== renamed file 'docs/administration/getting_started.rst' => 'docs/administration/index.rst'
--- docs/administration/getting_started.rst 2011-10-23 16:01:37 +0000
+++ docs/administration/index.rst 2012-03-18 00:36:19 +0000
@@ -1,5 +1,7 @@
1Getting Started1.. _administering_drizzle:
2===============2
3Administering Drizzle
4=====================
35
4Drizzle is a unique database server because most of its functionality is6Drizzle is a unique database server because most of its functionality is
5provided by plugins. In fact, the primary job of :program:`drizzled`,7provided by plugins. In fact, the primary job of :program:`drizzled`,
68
=== modified file 'docs/administration/plugins.rst'
--- docs/administration/plugins.rst 2012-03-03 02:42:42 +0000
+++ docs/administration/plugins.rst 2012-03-18 00:36:19 +0000
@@ -1,5 +1,7 @@
1.. program:: drizzled1.. program:: drizzled
22
3.. _plugins:
4
3Plugins5Plugins
4=======6=======
57
@@ -24,7 +26,7 @@
24 * :ref:`connection_id_plugin` - Return the current connection_id (connection_id)26 * :ref:`connection_id_plugin` - Return the current connection_id (connection_id)
25 * console_plugin - Console Client (console) (TODO: documentation missing)27 * console_plugin - Console Client (console) (TODO: documentation missing)
26 * :ref:`crc32_plugin` - CRC32 Function (crc32)28 * :ref:`crc32_plugin` - CRC32 Function (crc32)
27 * :ref:`default_replicator_plugin` - Default Replicator (default_replicator)29 * :ref:`default_replicator` - Default Replicator (default_replicator)
28 * :ref:`drizzle_protocol_plugin` - Drizzle Protocol (drizzle_protocol)30 * :ref:`drizzle_protocol_plugin` - Drizzle Protocol (drizzle_protocol)
29 * :ref:`errmsg_stderr_plugin` - Error Messages to stderr (errmsg_stderr)31 * :ref:`errmsg_stderr_plugin` - Error Messages to stderr (errmsg_stderr)
30 * :ref:`error_dictionary_plugin` - Data Dictionary for Errors. (error_dictionary)32 * :ref:`error_dictionary_plugin` - Data Dictionary for Errors. (error_dictionary)
3133
=== modified file 'docs/clients/drizzledump.rst'
--- docs/clients/drizzledump.rst 2012-03-08 00:54:48 +0000
+++ docs/clients/drizzledump.rst 2012-03-18 00:36:19 +0000
@@ -1,3 +1,5 @@
1.. _drizzledump:
2
1drizzledump Backup Tool3drizzledump Backup Tool
2=======================4=======================
35
46
=== modified file 'docs/configuration/drizzled.rst'
--- docs/configuration/drizzled.rst 2012-03-08 00:54:48 +0000
+++ docs/configuration/drizzled.rst 2012-03-18 00:36:19 +0000
@@ -15,6 +15,10 @@
15General Options15General Options
16^^^^^^^^^^^^^^^16^^^^^^^^^^^^^^^
1717
18.. option:: --daemon, -d
19
20 Run :program:`drizzled` as a daemon.
21
18.. option:: --help, -?22.. option:: --help, -?
1923
20 Display this help and exit.24 Display this help and exit.
@@ -23,6 +27,20 @@
2327
24 Display this help and exit after initializing plugins.28 Display this help and exit after initializing plugins.
2529
30.. option:: --user, -u ARG
31
32 :Default:
33 :Variable:
34
35 Run drizzled daemon as user.
36
37.. option:: --version, -V
38
39 :Default:
40 :Variable: ``version``
41
42 Print the version of Drizzle and exit.
43
26.. _drizzled_config_file_options:44.. _drizzled_config_file_options:
2745
28Config File Options46Config File Options
@@ -93,6 +111,25 @@
93111
94 --plugin_remove=syslog,md5112 --plugin_remove=syslog,md5
95113
114.. _drizzled_replication_options:
115
116Replication Options
117^^^^^^^^^^^^^^^^^^^
118
119.. option:: --replicate-query
120
121 :Default:
122 :Variable: ``replicate_query``
123
124 Include the SQL query in replicated protobuf messages.
125
126.. option:: --transaction-message-threshold
127
128 :Default: 1048576
129 :Variable: ``transaction_message_threshold``
130
131 Max message size written to transaction log, valid values 131072 - 1048576 bytes.
132
96.. _drizzled_kernel_options:133.. _drizzled_kernel_options:
97134
98Kernel Options135Kernel Options
@@ -421,13 +458,6 @@
421 A global constraint for read-rnd-buffer-size for all clients, cannot be set458 A global constraint for read-rnd-buffer-size for all clients, cannot be set
422 lower than --read-rnd-buffer-size. Setting to 0 means unlimited.459 lower than --read-rnd-buffer-size. Setting to 0 means unlimited.
423460
424.. option:: --replicate-query
425
426 :Default:
427 :Variable: ``replicate_query``
428
429 Include the SQL query in replicated protobuf messages.
430
431.. option:: --scheduler ARG461.. option:: --scheduler ARG
432462
433 :Default: multi-thread463 :Default: multi-thread
@@ -539,27 +569,6 @@
539569
540 Default transaction isolation level.570 Default transaction isolation level.
541571
542.. option:: --transaction-message-threshold
543
544 :Default: 1048576
545 :Variable: ``transaction_message_threshold``
546
547 Max message size written to transaction log, valid values 131072 - 1048576 bytes.
548
549.. option:: --user, -u ARG
550
551 :Default:
552 :Variable:
553
554 Run drizzled daemon as user.
555
556.. option:: --version, -V
557
558 :Default:
559 :Variable: ``version``
560
561 Output version information and exit.
562
563.. _drizzled_variables:572.. _drizzled_variables:
564573
565Variables574Variables
566575
=== modified file 'docs/configuration/options.rst'
--- docs/configuration/options.rst 2011-11-14 22:29:19 +0000
+++ docs/configuration/options.rst 2012-03-18 00:36:19 +0000
@@ -129,8 +129,8 @@
129^^^^^^^^^^^^^^^^^^^^129^^^^^^^^^^^^^^^^^^^^
130130
131Command line options have the form ``--option-name=value`` (the ``=`` is131Command line options have the form ``--option-name=value`` (the ``=`` is
132optional). This form works for both :ref:`drizzled_options` and all132optional). This form works for :ref:`drizzled options <drizzled_options>`
133plugin options. For example::133and all plugin options. For example::
134134
135 drizzled --basedir=/opt/drizzle --innodb.buffer-pool-size=500M135 drizzled --basedir=/opt/drizzle --innodb.buffer-pool-size=500M
136136
137137
=== modified file 'docs/functions/datetime.rst'
--- docs/functions/datetime.rst 2012-03-14 19:16:59 +0000
+++ docs/functions/datetime.rst 2012-03-18 00:36:19 +0000
@@ -29,7 +29,11 @@
29|statement_timestamp() |timestamp with time zone* |Current date and time (start of current statement) |29|statement_timestamp() |timestamp with time zone* |Current date and time (start of current statement) |
30+-----------------------------------+-------------------------------------+-----------------------------------------------------------------+30+-----------------------------------+-------------------------------------+-----------------------------------------------------------------+
3131
32<<<<<<< TREE
32Drizzle timezone is always UTC.33Drizzle timezone is always UTC.
34=======
35*Drizzle timezone is always UTC*.
36>>>>>>> MERGE-SOURCE
3337
34Extract38Extract
35-------39-------
3640
=== modified file 'docs/index.rst'
--- docs/index.rst 2012-01-30 21:09:19 +0000
+++ docs/index.rst 2012-03-18 00:36:19 +0000
@@ -73,7 +73,7 @@
73.. toctree::73.. toctree::
74 :maxdepth: 274 :maxdepth: 2
75 75
76 administration/getting_started76 administration/index
77 administration/drizzled77 administration/drizzled
78 administration/authentication78 administration/authentication
79 administration/authorization79 administration/authorization
@@ -85,8 +85,13 @@
85-----------85-----------
86.. toctree::86.. toctree::
87 :maxdepth: 287 :maxdepth: 2
88 :glob:
8889
89 replication/drizzle90 replication/index
91 replication/replicators/index
92 replication/appliers/index
93 replication/messages/index
94 replication/examples/index
9095
91SQL Language96SQL Language
92------------97------------
9398
=== added directory 'docs/replication/appliers'
=== added file 'docs/replication/appliers/index.rst'
--- docs/replication/appliers/index.rst 1970-01-01 00:00:00 +0000
+++ docs/replication/appliers/index.rst 2012-03-18 00:36:19 +0000
@@ -0,0 +1,33 @@
1.. _appliers:
2
3.. _replication_appliers:
4
5Appliers
6========
7
8Appliers are the other end point of a
9:ref:`replication stream <replication_streams>`, the first being
10:ref:`replicators`. Appliers provide an
11interface between replicators and a service. The service can be anything:
12another Drizzle server, a different database server, a message queue, etc.
13Appliers receive replication events from replicators and apply them to the
14service, although the term "apply" is used loosely. Appliers can do anything
15with replication events that provides useful behavior for the service.
16For example, an applier may write the replicaiton event to a file on disk,
17or it may send it over the network to some other service to be processed.
18
19Appliers are implemented by plugins and specify the unique name of a replicator with which Drizzle should pair it to create a replicaiton stream. Applier plugins default to using the :ref:`default_replicator`, but they can be configured to use another replicator.
20
21Most applier plugins are loaded and ran on the :ref:`originating_server`,
22but this is not a requirement. For example, the :ref:`slave_applier` is
23loaded on one server (the slave) and connects to and pairs with the
24:ref:`default_replicator` on another server (the master).
25
26Drizzle includes the following applier plugins:
27
28.. toctree::
29 :maxdepth: 1
30
31 slave
32 rabbitmq
33 zeromq
034
=== added directory 'docs/replication/appliers/rabbitmq'
=== added file 'docs/replication/appliers/rabbitmq.rst'
--- docs/replication/appliers/rabbitmq.rst 1970-01-01 00:00:00 +0000
+++ docs/replication/appliers/rabbitmq.rst 2012-03-18 00:36:19 +0000
@@ -0,0 +1,199 @@
1.. _rabbitmq_applier:
2
3RabbitMQ Applier
4================
5
6The RabbitMQ applier plugin, named ``rabbitmq``, applies replication events to a `RabbitMQ <http://www.rabbitmq.com>`_ server. This can be used to create advanced replication solutions, to visualize data, or to build triggers.
7
8Loading
9-------
10
11To load this plugin, start :program:`drizzled` with::
12
13 --plugin-add=rabbitmq
14
15Loading the plugin may not enable or configure it. See the plugin's
16:ref:`rabbitmq_configuration` and :ref:`rabbitmq_variables`.
17
18.. seealso:: :ref:`drizzled_plugin_options` for more information about adding and removing plugins.
19
20.. _rabbitmq_configuration:
21
22Configuration
23-------------
24
25These command line options configure the plugin when :program:`drizzled`
26is started. See :ref:`command_line_options` for more information about
27specifying command line options.
28
29.. program:: drizzled
30
31.. option:: --rabbitmq.exchange ARG
32
33 :Default: ReplicationExchange
34 :Variable: :ref:`rabbitmq_exchange <rabbitmq_exchange>`
35
36 Name of RabbitMQ exchange to publish to
37
38.. option:: --rabbitmq.host ARG
39
40 :Default: localhost
41 :Variable: :ref:`rabbitmq_host <rabbitmq_host>`
42
43 Host name to connect to
44
45.. option:: --rabbitmq.password ARG
46
47 :Default: guest
48 :Variable: :ref:`rabbitmq_password <rabbitmq_password>`
49
50 RabbitMQ password
51
52.. option:: --rabbitmq.port ARG
53
54 :Default: 5672
55 :Variable: :ref:`rabbitmq_port <rabbitmq_port>`
56
57 Port to connect to
58
59.. option:: --rabbitmq.routingkey ARG
60
61 :Default: ReplicationRoutingKey
62 :Variable: :ref:`rabbitmq_routingkey <rabbitmq_routingkey>`
63
64 Name of RabbitMQ routing key to use
65
66.. option:: --rabbitmq.use-replicator ARG
67
68 :Default: default_replicator
69 :Variable:
70
71 Name of the replicator plugin to use (default='default_replicator')
72
73.. option:: --rabbitmq.username ARG
74
75 :Default: guest
76 :Variable: :ref:`rabbitmq_username <rabbitmq_username>`
77
78 RabbitMQ username
79
80.. option:: --rabbitmq.virtualhost ARG
81
82 :Default: /
83 :Variable: :ref:`rabbitmq_virtualhost <rabbitmq_virtualhost>`
84
85 RabbitMQ virtualhost
86
87.. _rabbitmq_variables:
88
89Variables
90---------
91
92These variables show the running configuration of the plugin.
93See `variables` for more information about querying and setting variables.
94
95.. _rabbitmq_exchange:
96
97* ``rabbitmq_exchange``
98
99 :Scope: Global
100 :Dynamic: No
101 :Option: :option:`--rabbitmq.exchange`
102
103 Name of RabbitMQ exchange to publish to
104
105.. _rabbitmq_host:
106
107* ``rabbitmq_host``
108
109 :Scope: Global
110 :Dynamic: No
111 :Option: :option:`--rabbitmq.host`
112
113 Host name to connect to
114
115.. _rabbitmq_password:
116
117* ``rabbitmq_password``
118
119 :Scope: Global
120 :Dynamic: No
121 :Option: :option:`--rabbitmq.password`
122
123 RabbitMQ password
124
125.. _rabbitmq_port:
126
127* ``rabbitmq_port``
128
129 :Scope: Global
130 :Dynamic: No
131 :Option: :option:`--rabbitmq.port`
132
133 Port to connect to
134
135.. _rabbitmq_routingkey:
136
137* ``rabbitmq_routingkey``
138
139 :Scope: Global
140 :Dynamic: No
141 :Option: :option:`--rabbitmq.routingkey`
142
143 Name of RabbitMQ routing key to use
144
145.. _rabbitmq_username:
146
147* ``rabbitmq_username``
148
149 :Scope: Global
150 :Dynamic: No
151 :Option: :option:`--rabbitmq.username`
152
153 RabbitMQ username
154
155.. _rabbitmq_virtualhost:
156
157* ``rabbitmq_virtualhost``
158
159 :Scope: Global
160 :Dynamic: No
161 :Option: :option:`--rabbitmq.virtualhost`
162
163 RabbitMQ virtualhost
164
165
166Implementation Details
167----------------------
168
169* :program:`drizzled` will not sart if the rabbitmq server is not available.
170* If the rabbitmq server goes away, the plugin will try to reconnect and resend the message 3 times, after that, the transaction is rolled back.
171
172.. _rabbitmq_version:
173
174Version
175-------
176
177This documentation applies to :ref:`rabbitmq_0.1_drizzle_7.0`.
178
179To see which version of the ``rabbitmq`` plugin a Drizzle server is running,
180execute:
181
182.. code-block:: mysql
183
184 SELECT MODULE_VERSION FROM DATA_DICTIONARY.MODULES WHERE MODULE_NAME='rabbitmq'
185
186Changelog
187---------
188
189.. toctree::
190 :maxdepth: 2
191
192 rabbitmq/changelog
193
194.. _rabbitmq_authors:
195
196Authors
197-------
198
199Marcus Eriksson
0200
=== added file 'docs/replication/appliers/rabbitmq/changelog.rst'
--- docs/replication/appliers/rabbitmq/changelog.rst 1970-01-01 00:00:00 +0000
+++ docs/replication/appliers/rabbitmq/changelog.rst 2012-03-18 00:36:19 +0000
@@ -0,0 +1,13 @@
1.. _rabbitmq_changelog:
2
3rabbitmq Changelog
4==================
5
6.. _rabbitmq_0.1_drizzle_7.0:
7
8rabbitmq 0.1 (Drizzle 7.0)
9--------------------------
10
11:Released: March 15, 2011
12
13* First release.
014
=== added directory 'docs/replication/appliers/slave'
=== added file 'docs/replication/appliers/slave.rst'
--- docs/replication/appliers/slave.rst 1970-01-01 00:00:00 +0000
+++ docs/replication/appliers/slave.rst 2012-03-18 00:36:19 +0000
@@ -0,0 +1,64 @@
1.. _slave:
2
3.. _slave_applier:
4
5Slave Applier
6=============
7
8The slave applier plugin, named ``slave``, provides a native implementation
9of replication between Drizzle servers by applying replication events from
10multiple "master" servers to a "slave" server. A slave is a server on which
11the ``slave`` plugin is loaded, configured, and ran. Slaves can connect to up
12to ten masters at once over the network and pair with the
13:ref:`default_replicator` on each them. Slaves apply every replication
14event from every master; conequently, they contain the same schemas and
15data as their masters.
16
17Replication chains are supported: slaves can also be masters and replicate
18events to other slaves. There is no limit to the number of levels in which
19masters and slaves can be enchained.
20
21Replication circles, or master-master replication, are not currently supported.
22Replication will break on one of the masters if this is attempted.
23
24The next two sections, :ref:`slave_configuration` and :ref:`slave_administration`, contain necessary information for configuring and administering slave-based replication. The third section, :ref:`slave_details`, covers technical details about how the ``slave`` plugin and slave-based replication work.
25
26.. toctree::
27 :maxdepth: 2
28
29 slave/configuration
30 slave/administration
31 slave/details
32
33.. _slave_version:
34
35Version
36-------
37
38This documentation applies to :ref:`slave_1.1_drizzle_7.1`.
39
40To see which version of the ``slave`` plugin a Drizzle server is running,
41execute:
42
43.. code-block:: mysql
44
45 SELECT MODULE_VERSION FROM DATA_DICTIONARY.MODULES WHERE MODULE_NAME='slave'
46
47Changelog
48---------
49
50.. toctree::
51 :maxdepth: 2
52
53 slave/changelog
54
55.. _slave_authors:
56
57Authors
58-------
59
60David Shrewsbury
61 Original ``slave`` plugin code. Multi-master code.
62
63Daniel Nichter
64 Documentation for :ref:`slave_1.1_drizzle_7.1`.
065
=== added file 'docs/replication/appliers/slave/administration.rst'
--- docs/replication/appliers/slave/administration.rst 1970-01-01 00:00:00 +0000
+++ docs/replication/appliers/slave/administration.rst 2012-03-18 00:36:19 +0000
@@ -0,0 +1,332 @@
1.. _slave_administration:
2
3.. _slave_admin:
4
5Slave Administration
6********************
7
8.. _slave_threads:
9
10Slave Threads
11=============
12
13The :ref:`slave` plugin uses two threads per master to read and apply replication events:
14
15* An IO (or producer) thread
16* An applier (or consumer) thread
17
18The IO thread connects to a master and reads replication events from the :ref:`slave_innodb_transaction_log` on the master. The applier thread applies the replication events from the IO thread to the slave. In other words, the IO thread produces replication events from the master, and the applier thread consumes and applies them to the slave.
19
20There is currently no way to stop, start, or restart the threads. Unless a replication error occurs, the threads should function continuously and remain in :ref:`RUNNING <slave_thread_running_status>` status. If a replication error does occur, the Drizzle server must be stopped, the error fixed, and the server restarted.
21
22.. _slave_thread_statuses:
23
24Statuses
25--------
26
27Slave thread statuses are indicated by the ``status`` columns of the :ref:`sys_replication_tables`. A slave thread status is always one of these values:
28
29.. _slave_thread_running_status:
30
31RUNNING
32 The thread is working properly, applying replicaiton events.
33
34.. _slave_thread_stopped_status:
35
36STOPPED
37 The thread has stopped due to an error; replication events are not being applied.
38
39.. _sys_replication_tables:
40
41sys_replication Tables
42======================
43
44The ``sys_replication`` schema on the slave has tables with information about the state of :ref:`slave_threads` and other replication-related information:
45
46.. code-block:: mysql
47
48 drizzle> SHOW TABLES FROM sys_replication;
49 +---------------------------+
50 | Tables_in_sys_replication |
51 +---------------------------+
52 | applier_state |
53 | io_state |
54 | queue |
55 +---------------------------+
56
57If the ``sys_replication`` table is not available, then the :ref:`slave` plugin failed to load.
58
59.. _sys_replication_io_state:
60
61io_state
62--------
63
64``sys_replication.io_state`` contains the state of IO threads, one per row:
65
66.. code-block:: mysql
67
68 drizzle> SELECT * FROM sys_replication.io_state;
69 *************************** 1. row ***************************
70 master_id: 1
71 status: RUNNING
72 error_msg:
73
74master_id
75 ID (number) of the master to which the thread is connected. The number corresponds to the the master number in the :ref:`slave_config_file`.
76
77status
78 :ref:`Status <slave_thread_statuses>` of the IO thread.
79
80error_msg
81 Error message explaining why the thread has :ref:`STOPPED <slave_thread_statuses>`.
82
83.. _sys_replication_applier_state:
84
85applier_state
86-------------
87
88``sys_replication.applier_state`` contains the state of applier threads, one per row:
89
90.. code-block:: mysql
91
92 drizzle> SELECT * FROM sys_replication.applier_state\G
93 *************************** 1. row ***************************
94 master_id: 1
95 last_applied_commit_id: 18
96 originating_server_uuid: 9908C6AA-A982-4763-B9BA-4EF5F933D219
97 originating_commit_id: 18
98 status: RUNNING
99 error_msg:
100
101master_id
102 ID (number) of the master from which the thread is applying replication events. The number corresponds to the the master number in the :ref:`slave_config_file`.
103
104last_applied_commit_id
105 Value of the ``COMMIT_ID`` from the master's replication log of the most recently executed transaction. See definition of the data_dictionary.sys_replication_log table.
106
107originating_server_uuid
108 UUID of the :ref:`originating_server`.
109
110originating_commit_id
111 ``COMMIT_ID`` from the :ref:`originating_server`.
112
113status
114 :ref:`Status <slave_thread_statuses>` of the applier thread.
115
116error_msg
117 Error message explaining why the thread has :ref:`STOPPED <slave_thread_stopped_status>`.
118
119.. _sys_replication_queue:
120
121queue
122-----
123
124``sys_replication.io_state`` contains replication events that have not yet been applied by the applier thread, one per row:
125
126.. code-block:: mysql
127
128 drizzle> SELECT * FROM sys_replication.queue\G
129 *************************** 1. row ***************************
130 trx_id: 925
131 seg_id: 1
132 commit_order: 12
133 originating_server_uuid: 9908C6AA-A982-4763-B9BA-4EF5F933D219
134 originating_commit_id: 12
135 msg: transaction_context {
136 server_id: 1
137 transaction_id: 925
138 start_timestamp: 1330211976689868
139 end_timestamp: 1330211976689874
140 }
141 statement {
142 type: DROP_SCHEMA
143 start_timestamp: 1330211976689872
144 end_timestamp: 1330211976689873
145 drop_schema_statement {
146 schema_name: "foo"
147 }
148 }
149 segment_id: 1
150 end_segment: true
151 master_id: 1
152
153.. _slave_lag:
154
155Slave Lag
156=========
157
158Slaves intentionally lag masters by at least :ref:`applier-thread-sleep <slave_cfg_common_options>` or :ref:`io-thread-sleep <slave_cfg_common_options>` seconds. Since :ref:`replication_events` are transactions, *slave lag is the number of committed transactions behind a master*. To determine a slave's lag, first query the last committed transaction ID on the master:
159
160.. code-block:: mysql
161
162 drizzle> SELECT MAX(COMMIT_ID) FROM DATA_DICTIONARY.SYS_REPLICATION_LOG;
163 +----------------+
164 | MAX(COMMIT_ID) |
165 +----------------+
166 | 6 |
167 +----------------+
168
169Then query the last applied transaction ID on the slave:
170
171.. code-block:: mysql
172
173 drizzle> SELECT MAX(last_applied_commit_id) FROM sys_replication.applier_state;
174 +-----------------------------+
175 | MAX(last_applied_commit_id) |
176 +-----------------------------+
177 | 6 |
178 +-----------------------------+
179
180In this example, the slave is caught up to the master because both the last commited transaction ID on the master and the last applied transaction ID on the slave are 6.
181
182If the last applied transaction ID on a slave is less than the last committed transaciton ID on a master, then the slave lags the master by *N transactions* (not seconds) where N is the difference of the two transaction ID values.
183
184Master Connections
185==================
186
187Slaves connect to masters like normal users by specifying a username and password (see the :ref:`slave_cfg_master_options`). Therefore, slave connections on a master are visible in the master's processlist and sessions, but they are not specially indicated. If the :ref:`slave_user_account` uses slave-specific usernames like "slave1", then the slave connections can be viewed like:
188
189.. code-block:: mysql
190
191 drizzle> SELECT * FROM DATA_DICTIONARY.PROCESSLIST WHERE USERNAME LIKE 'slave%'\G
192 *************************** 1. row ***************************
193 ID: 2
194 USERNAME: slave
195 HOST: 127.0.0.1
196 DB: NULL
197 COMMAND: Sleep
198 TIME: 0
199 STATE: NULL
200 INFO: NULL
201 HAS_GLOBAL_LOCK: 0
202
203The ``DATA_DICTIONARY.SESSIONS`` table can be queried similarly:
204
205.. code-block:: mysql
206
207 drizzle> SELECT * FROM DATA_DICTIONARY.SESSIONS WHERE SESSION_USERNAME LIKE 'slave%'\G
208 *************************** 1. row ***************************
209 SESSION_ID: 2
210 SESSION_USERNAME: slave1
211 SESSION_HOST: 127.0.0.1
212 SESSION_CATALOG: LOCAL
213 SESSION_SCHEMA: NULL
214 COMMAND: Sleep
215 STATE: NULL
216 QUERY: NULL
217 HAS_GLOBAL_LOCK: 0
218 IS_INTERACTIVE: 0
219 IS_CONSOLE: 0
220
221Or, slave connections can be viewed by specifying the slave server's hostname, like:
222
223.. code-block:: mysql
224
225 drizzle> SELECT * FROM DATA_DICTIONARY.PROCESSLIST WHERE HOSTNAME = '192.168.1.5'\G
226 *************************** 1. row ***************************
227 ID: 3
228 USERNAME: slave
229 HOST: 192.168.1.5
230 DB: NULL
231 COMMAND: Sleep
232 TIME: 0
233 STATE: NULL
234 INFO: NULL
235 HAS_GLOBAL_LOCK: 0
236
237.. _slave_innodb_transaction_log:
238
239InnoDB Transaction Log
240======================
241
242The :ref:`slave` requires the :ref:`innodb_transaction_log` on the master to retrieve replication events. This transaction log is stored as an internal table within InnoDB, but there are two tables which provide access to its data:
243
244* ``DATA_DICTIONARY.SYS_REPLICATION_LOG``
245* ``DATA_DICTIONARY.INNODB_REPLICATION_LOG``
246
247The :ref:`IO thread <slave_threads>` from a slave (which connects to a master) reads transactions (replicaiton events) directly from ``DATA_DICTIONARY.SYS_REPLICATION_LOG``. The transaction messages are binary which makes the table data unreadable by most humans:
248
249.. code-block:: mysql
250
251 drizzle> SELECT * from DATA_DICTIONARY.SYS_REPLICATION_LOG\G
252 *************************** 1. row ***************************
253 ID: 772
254 SEGID: 1
255 COMMIT_ID: 1
256 END_TIMESTAMP: 1331841800496546
257 ORIGINATING_SERVER_UUID: 98ECEA09-BA65-489D-9382-F8D15098B1AE
258 ORIGINATING_COMMIT_ID: 1
259 MESSAGE_LEN: 33
260 MESSAGE: ?????????
261
262The last column, ``MESSAGE``, contains the actual transaction data that the client renders as question marks because the data is binary, not text.
263
264The ``DATA_DICTIONARY.INNODB_REPLICATION_LOG`` table contains the same data as the ``DATA_DICTIONARY.SYS_REPLICATION_LOG`` table, but it converts the transaction data to text:
265
266.. code-block:: mysql
267
268 drizzle> SELECT * from DATA_DICTIONARY.INNODB_REPLICATION_LOG\G
269 *************************** 1. row ***************************
270 TRANSACTION_ID: 772
271 TRANSACTION_SEGMENT_ID: 1
272 COMMIT_ID: 1
273 END_TIMESTAMP: 1331841800496546
274 ORIGINATING_SERVER_UUID: 98ECEA09-BA65-489D-9382-F8D15098B1AE
275 ORIGINATING_COMMIT_ID: 1
276 TRANSACTION_MESSAGE_STRING: transaction_context {
277 server_id: 1
278 transaction_id: 772
279 start_timestamp: 1331841800496542
280 end_timestamp: 1331841800496546
281 }
282 event {
283 type: STARTUP
284 }
285 segment_id: 1
286 end_segment: true
287 TRANSACTION_LENGTH: 33
288
289The ``TRANSACTION_MESSAGE_STRING`` column contains the text representation of the ``MESSAGE`` column from the ``DATA_DICTIONARY.SYS_REPLICATION_LOG`` table.
290
291
292``DATA_DICTIONARY.INNODB_REPLICATION_LOG`` is read-only, but ``DATA_DICTIONARY.SYS_REPLICATION_LOG`` can be modified which allows the transaction log to be maintained, as described in the next section.
293
294Transaction Log Maintenance
295---------------------------
296
297Currently, the InnoDB transaction log grows without bounds and old transactions are never deleted. The InnoDB transaction log must be maintained manually by carefully deleting old transactions that are no longer needed from the ``DATA_DICTIONARY.SYS_REPLICATION_LOG`` table.
298
299.. warning:: Care must be taken to avoid deleting transactions that slaves have not yet applied else data will be lost and replication will break.
300
301Follow these steps to trim the InnoDB transaction log without affecting slave function:
302
303#. Query each slave for the ``last_applied_commit_id`` value from the :ref:`sys_replication.applier_state <sys_replication_applier_state>` table.
304#. Choose the **minimum** value obtained from step one. This is the marker value for the slave that is the furthest behind the master.
305#. Using the marker value from the previous step, delete rows from ``DATA_DICTIONARY.SYS_REPLICATION_LOG`` that have a ``COMMIT_ID`` less than the marker value.
306
307For example, if there are two slaves, query each one for the minimum ``last_applied_commit_id``:
308
309.. code-block:: mysql
310
311 slave1> SELECT last_applied_commit_id FROM sys_replicaiton.applier_state;
312 +------------------------+
313 | last_applied_commit_id |
314 +------------------------+
315 | 3000 |
316 +------------------------+
317
318 slave2> SELECT last_applied_commit_id FROM sys_replicaiton.applier_state;
319 +------------------------+
320 | last_applied_commit_id |
321 +------------------------+
322 | 2877 |
323 +------------------------+
324
325slave2 has the smallest value for ``last_applied_commit_id``, 2877, so this value is the marker for deleting records from the master's transaction log:
326
327.. code-block:: mysql
328
329 master> DELETE FROM DATA_DICTIONARY.SYS_REPLICATION_LOG
330 -> WHERE COMMIT_ID < 2877;
331
332This permanently deletes all old, unneeded records from the InnoDB transaction log.
0333
=== added file 'docs/replication/appliers/slave/changelog.rst'
--- docs/replication/appliers/slave/changelog.rst 1970-01-01 00:00:00 +0000
+++ docs/replication/appliers/slave/changelog.rst 2012-03-18 00:36:19 +0000
@@ -0,0 +1,27 @@
1.. _slave_changelog:
2
3slave Changelog
4===============
5
6.. _slave_1.1_drizzle_7.1:
7
8slave 1.1 (Drizzle 7.1)
9-----------------------
10
11:Released:
12
13* Added multi-master-support (https://blueprints.launchpad.net/drizzle/+spec/slave-multi-master).
14
15 * Changed all options, except ``--slave.config-file``, from command line options to slave config file options.
16 * Added ``ignore-errors`` option for slave config file.
17
18* Rewrote :ref:`slave` docs (https://blueprints.launchpad.net/drizzle/+spec/docs71-focus-areas).
19
20.. _slave_1.0_drizzle_7.0:
21
22slave 1.0 (Drizzle 7.0)
23-----------------------
24
25:Released: March 15, 2011
26
27* First release.
028
=== added file 'docs/replication/appliers/slave/configuration.rst'
--- docs/replication/appliers/slave/configuration.rst 1970-01-01 00:00:00 +0000
+++ docs/replication/appliers/slave/configuration.rst 2012-03-18 00:36:19 +0000
@@ -0,0 +1,229 @@
1.. program:: drizzled
2
3.. _slave_configuration:
4
5.. _slave_config:
6
7Slave Configuration
8*******************
9
10Configuring a Drizzle :ref:`replication stream <replication_streams>` using the :ref:`slave` requires:
11
12#. Load a :ref:`replicator plugin <replicators>` on the masters
13#. Enabling :option:`--innodb.replication-log` on the masters
14#. Creating user accounts for the slaves on the masters
15#. Writing a :ref:`slave_config_file` for :option:`--slave.config-file` on the slave
16#. Loading and configure the ``slave`` plugin on the slave
17
18The first three steps are performed on a master, and the last two steps are performed on a slave.
19
20.. _configuring_a_master:
21
22Configuring a Master
23====================
24
25A single slave can apply replication events from up to ten masters at once.
26There are three requirements for each master:
27
28#. :ref:`replicators`
29#. :option:`--innodb.replication-log`
30#. :ref:`slave_user_account`
31
32A :ref:`replicator plugin <replicators>` must be loaded on each master. The :ref:`default_replicator` is loaded by default. To verify that it is loaded on a master, execute:
33
34.. code-block:: sql
35
36 drizzle> SELECT * FROM DATA_DICTIONARY.PLUGINS WHERE PLUGIN_NAME = 'default_replicator';
37 +--------------------+-----------------------+-----------+--------------------+
38 | PLUGIN_NAME | PLUGIN_TYPE | IS_ACTIVE | MODULE_NAME |
39 +--------------------+-----------------------+-----------+--------------------+
40 | default_replicator | TransactionReplicator | 1 | default_replicator |
41 +--------------------+-----------------------+-----------+--------------------+
42
43If the plugin is not loaded, verify that the server was *not* started with
44``--plugin-remove default_replicator``. If it was, remove that option and
45restart the server.
46
47To use a different replicator, :ref:`configure Drizzle <configuring_drizzle>` to load the replicator plugin on startup, and specify the replicator name with :option:`--innodb.use-replicator`.
48
49Each master must also be started with :option:`--innodb.replication-log`
50to enable the :ref:`innodb_transaction_log` which is not enabled by default.
51Therefore, Drizzle must be configured with this option at startup.
52See :ref:`configuring_drizzle` for more information. To verify that the
53InnoDB replication log is active, execute:
54
55.. code-block:: mysql
56
57 drizzle> SELECT * FROM DATA_DICTIONARY.GLOBAL_VARIABLES WHERE VARIABLE_NAME = 'innodb_replication_log';
58 +------------------------+----------------+
59 | VARIABLE_NAME | VARIABLE_VALUE |
60 +------------------------+----------------+
61 | innodb_replication_log | ON |
62 +------------------------+----------------+
63
64 drizzle> SELECT * FROM DATA_DICTIONARY.INNODB_REPLICATION_LOG LIMIT 1;
65 -- The query should return one row showing a replication event.
66
67.. _slave_user_account:
68
69Slave User Account
70------------------
71
72A user account is required on the master for slave connections, unless no :ref:`authentication` is used (which is highly inadvisable). One user account can be used for all slaves, or individual user accounts can be used for each slave. In either case, the user account credentials (username and password) for a master are specified in the :ref:`slave_config_file`.
73
74:ref:`authorization` must be configured on the master to allow slave user accounts to access the ``DATA_DICTIONARY`` schema, else the :ref:`slave IO thread <slave_threads>` will fail to start on the slave:
75
76.. code-block:: mysql
77
78 drizzle> SELECT * FROM sys_replication.io_state\G
79 *************************** 1. row ***************************
80 master_id: 1
81 status: STOPPED
82 error_msg: Replication slave: Access denied for user 'slave' to schema 'data_dictionary'
83
84The :ref:`sys_replication_tables` are discussed in :ref:`slave_admin`.
85
86:ref:`authorization` cannot be used to filter which schemas or tables are replicated because slave connections do no access individual schemas and tables, they only access the :ref:`innodb_transaction_log`. To filter which schemas or tables are replicated, configure the master to use a filtering replicator like :ref:`filtered_replicator`, as described above.
87
88.. _configuring_a_slave:
89
90Configuring a Slave
91===================
92
93After :ref:`configuring_a_master`, configuring a slave requires only:
94
95#. :ref:`slave_config_file`
96#. :ref:`slave_plugin`
97
98.. _slave_config_file:
99
100Slave Config File
101-----------------
102
103A slave config file is a plain text file that contains connection and configuration options for each master. At least one master must be specifed, and masters must be numbered sequentially from 1 to 10. The general syntax of a slave config file is:
104
105.. code-block:: ini
106
107 # comment
108 common-option=value
109 [masterN]
110 master-specific-option=value
111
112There are two types of options: common options which apply to all masters, and master-specific options which only apply to the preceding ``[masterN]`` header where ``N`` is the sequentially numbered master, starting with 1. Whitespace
113before and after lines and around ``=`` (equal signs) is ignored.
114
115The simplest possible slave config file is:
116
117.. code-block:: ini
118
119 [master1]
120 master-host=<master hostname>
121 master-user=slave1
122
123See :ref:`slave_examples` for complete, working examples.
124
125.. _slave_cfg_common_options:
126
127Common Options
128--------------
129
130These options must be specified first, before any ``[masterN]`` headers.
131
132.. confval:: applier-thread-sleep
133
134 :Default: 5
135
136 The number of seconds the applier (consumer) thread sleeps between applying
137 replication events from the local queue.
138
139.. confval:: ignore-errors
140
141 Ignore errors and continue applying replication events. It is generally
142 a bad idea to use this option!
143
144.. confval:: io-thread-sleep
145
146 :Default: 5
147
148 The number of seconds the IO (producer) thread sleeps between queries to the
149 master for more replication events.
150
151.. confval:: seconds-between-reconnects
152
153 :Default: 30
154
155 The number of seconds to wait between reconnect attempts when the master
156 server becomes unreachable.
157
158.. _slave_cfg_master_options:
159
160Master-specific Options
161-----------------------
162
163These options must be specified after a ``[masterN]`` header.
164
165.. confval:: master-host
166
167 Hostname/IP address of the master server.
168
169.. confval:: master-port
170
171 :Default: 3306
172
173 Drizzle port used by the master server.
174
175.. confval:: master-user
176
177 Username to use for connecting to the master server.
178 See :ref:`slave_user_account`.
179
180.. confval:: master-pass
181
182 Password associated with the username given by :confval:`master-user`.
183 See :ref:`slave_user_account`.
184
185.. program:: drizzledump
186
187.. confval:: max-commit-id
188
189 Maximum commit ID the slave is assumed to have applied from the master.
190 This value will be used by the slave to determine where to begin retrieving
191 replication events from the master transaction log. This option can be used
192 to provision a new slave by setting it to the value output from the
193 :ref:`drizzledump` when used with the :option:`--single-transaction` option.
194
195.. confval:: max-reconnects
196
197 :Default: 10
198
199 The number of reconnection attempts the slave plugin will try if the
200 master server becomes unreachable.
201
202.. _slave_plugin:
203
204slave Plugin
205------------
206
207A slave must load the ``slave`` plugin which is not loaded by default.
208This plugin has only one option:
209
210.. program:: drizzled
211
212.. option:: --slave.config-file FILE
213
214 :Default: :file:`BASEDIR/etc/slave.cfg`
215 :Variable:
216
217 Full path to a :ref:`slave_config_file`.
218 By default, the plugin looks for a file named :file:`slave.cfg`
219 in :file:`BASEDIR/etc/` where :file:`BASEDIR` is determined by
220 :option:`--basedir`.
221
222Since a slave can connect to multiple masters, all other options are set
223in the :ref:`slave_config_file`. Once a slave config file has been written, start Drizzle with the ``slave`` plugin like:
224
225.. code-block:: bash
226
227 $ drizzled --plugin-add slave --slave.config-file /etc/drizzled/slave.conf
228
229If the masters are configured properly, and the slave config file is correct, and the slave plugin loads successfully, the :ref:`sys_replication_tables` will be accessible on the slave as described in the next topic, :ref:`slave_admin`.
0230
=== added file 'docs/replication/appliers/slave/details.rst'
--- docs/replication/appliers/slave/details.rst 1970-01-01 00:00:00 +0000
+++ docs/replication/appliers/slave/details.rst 2012-03-18 00:36:19 +0000
@@ -0,0 +1,11 @@
1.. _slave_details:
2
3Slave Details
4=============
5
6The following sections dive into the technical aspects of the :ref:`slave`. In general, it should not be necessary to know this information to :ref:`configure <slave_configuration>` or :ref:`adminster <slave_admin>` a slave. This information is useful for troubleshooting, developers, hackers, and those who wish to "look under the hood."
7
8Slave Plugin Class
9------------------
10
11Although the documentation for the :ref:`slave` calls the plugin an applier, which implies that the plugin subclasses the TransactionApplier class, the :ref:`slave_plugin` is not in fact a TransactionApplier, it is a Daemon. The :ref:`innodb_transaction_log` is a TransactionApplier which defaults to using with the :ref:`default_replicator` (see :file:`plugin/innobase/handler/replication_log.h`).
012
=== added file 'docs/replication/appliers/zeromq.rst'
--- docs/replication/appliers/zeromq.rst 1970-01-01 00:00:00 +0000
+++ docs/replication/appliers/zeromq.rst 2012-03-18 00:36:19 +0000
@@ -0,0 +1,55 @@
1.. _zeromq:
2
3.. _zeromq_applier:
4
5ZeroMQ Applier
6==============
7
8ZeroMQ is a messaging library that allows you to easily build complex
9communication systems. The ZeroMQ plugin allows drizzle to publish
10transactions to a local PUB socket. Many clients can subscribe to
11these transactions. The first frame of the message sent out is the
12schema name the transaction touched - this enables clients to only
13subscribe to the interesting schemas (note that certain "transactions"
14are without a schema, like SET xyz for example, for these, the first
15frame is empty).
16
17Getting started
18---------------
19
20First, install zeromq, get the code from `zeromq.org
21<http://zeromq.org>`_, then you can build drizzle, watch the
22./configure output to verify that drizzle finds the libraries needed.
23
24Now you are good to go, simply start drizzle with --plugin-add=zeromq
25and drizzle will start publishing transactions. The only configuration
26parameter available is:
27
28.. code-block:: bash
29
30 --zeromq.endpoint arg (=tcp://*:9999) - the endpoint to expose.
31
32Now you can write a simple python script to verify that it works,
33something like this will work:
34
35.. code-block:: python
36
37 import zmq
38
39 ctx = zmq.Context()
40 s = ctx.socket(zmq.SUB)
41 s.setsockopt(zmq.SUBSCRIBE, '')
42 s.connect('tcp://localhost:9999')
43 i = 0
44 while True:
45 i = i+1
46 s.recv_multipart()
47 print i
48
49and then you can generate some load:
50
51.. code-block:: bash
52
53 bin/drizzleslap -c 10 -a --auto-generate-sql-add-autoincrement --burnin
54
55which creates 10 threads and generates random queries.
056
=== removed file 'docs/replication/drizzle.rst'
--- docs/replication/drizzle.rst 2011-10-23 05:45:09 +0000
+++ docs/replication/drizzle.rst 1970-01-01 00:00:00 +0000
@@ -1,324 +0,0 @@
1*******************
2Drizzle Replication
3*******************
4
5Replication events are recorded using messages in the `Google Protocol Buffer
6<http://code.google.com/p/protobuf/>`_ (GPB) format. GPB messages can contain
7sub-messages. There is a single main "envelope" message, Transaction, that
8is passed to plugins that subscribe to the replication stream.
9
10Configuration Options
11#####################
12
13**transaction_message_threshold**
14
15 Controls the size, in bytes, of the Transaction messages. When a Transaction
16 message exceeds this size, a new Transaction message with the same
17 transaction ID will be created to continue the replication events.
18 See :ref:`bulk-operations` below.
19
20
21**replicate_query**
22
23 Controls whether the originating SQL query will be included within each
24 Statement message contained in the enclosing Transaction message. The
25 default global value is FALSE which will not include the query in the
26 messages. It can be controlled per session, as well. For example:
27
28 .. code-block:: mysql
29
30 drizzle> set @@replicate_query = 1;
31
32 The stored query should be used as a guide only, and never executed
33 on a slave to perform replication as this will lead to incorrect results.
34
35Message Definitions
36###################
37
38The GPB messages are defined in .proto files in the drizzled/message
39directory of the Drizzle source code. The primary definition file is
40transaction.proto. Messages defined in this file are related in the
41following ways::
42
43
44 ------------------------------------------------------------------
45 | |
46 | Transaction message |
47 | |
48 | ----------------------------------------------------------- |
49 | | | |
50 | | TransactionContext message | |
51 | | | |
52 | ----------------------------------------------------------- |
53 | ----------------------------------------------------------- |
54 | | | |
55 | | Statement message 1 | |
56 | | | |
57 | ----------------------------------------------------------- |
58 | ----------------------------------------------------------- |
59 | | | |
60 | | Statement message 2 | |
61 | | | |
62 | ----------------------------------------------------------- |
63 | ... |
64 | ----------------------------------------------------------- |
65 | | | |
66 | | Statement message N | |
67 | | | |
68 | ----------------------------------------------------------- |
69 ------------------------------------------------------------------
70
71with each Statement message looking like so::
72
73 ------------------------------------------------------------------
74 | |
75 | Statement message |
76 | |
77 | ----------------------------------------------------------- |
78 | | | |
79 | | Common information | |
80 | | | |
81 | | - Type of Statement (INSERT, DELETE, etc) | |
82 | | - Start Timestamp | |
83 | | - End Timestamp | |
84 | | - (OPTIONAL) Actual SQL query string | |
85 | | | |
86 | ----------------------------------------------------------- |
87 | ----------------------------------------------------------- |
88 | | | |
89 | | Statement subclass message 1 (see below) | |
90 | | | |
91 | ----------------------------------------------------------- |
92 | ... |
93 | ----------------------------------------------------------- |
94 | | | |
95 | | Statement subclass message N (see below) | |
96 | | | |
97 | ----------------------------------------------------------- |
98 ------------------------------------------------------------------
99
100The Transaction Message
101#######################
102
103The main "envelope" message which represents an atomic transaction
104which changed the state of a server is the Transaction message class.
105
106The Transaction message contains two pieces:
107
108#. A TransactionContext message containing information about the
109 transaction as a whole, such as the ID of the executing server,
110 the start and end timestamp of the transaction, segmenting
111 metadata and a unique identifier for the transaction.
112#. A vector of Statement messages representing the distinct SQL
113 statements which modified the state of the server. The Statement
114 message is, itself, a generic envelope message containing a
115 sub-message which describes the specific data modification which
116 occurred on the server (such as, for instance, an INSERT statement).
117
118The Statement Message
119#####################
120
121The generic "envelope" message containing information common to each
122SQL statement executed against a server (such as a start and end timestamp
123and the type of the SQL statement) as well as a Statement subclass message
124describing the specific data modification event on the server.
125
126Each Statement message contains a type member which indicates how readers
127of the Statement should construct the inner Statement subclass representing
128a data change.
129
130Statements are recorded separately as sometimes individual statements
131have to be rolled back.
132
133
134.. _bulk-operations:
135
136How Bulk Operations Work
137########################
138
139Certain operations which change large volumes of data on a server
140present a specific set of problems for a transaction coordinator or
141replication service. If all operations must complete atomically on a
142publishing server before replicas are delivered the complete
143transactional unit:
144
145#. The publishing server could consume a large amount of memory
146 building an in-memory Transaction message containing all the
147 operations contained in the entire transaction.
148#. A replica, or subscribing server, is wasting time waiting on the
149 eventual completion (commit) of the large transaction on the
150 publishing server. It could be applying pieces of the large
151 transaction in the meantime...
152
153In order to prevent the problems inherent in (1) and (2) above, Drizzle's
154replication system uses a mechanism which provides bulk change
155operations.
156
157A single transaction in the database can possibly be represented with
158multiple protobuf Transaction messages if the message grows too large.
159This can happen if you have a bulk transaction, or a single statement
160affecting a very large number of rows, or just a large transaction with
161many statements/changes.
162
163For the first two examples, it is likely that the Statement sub-message
164itself will get segmented, causing another Transaction message to be
165created to hold the rest of the Statement's row changes. In these cases,
166it is enough to look at the segment information stored in the Statement
167message (see example below).
168
169For the last example, the Statement sub-messages may or may not be
170segmented, but we could still need to split the individual Statements up into
171multiple Transaction messages to keep the Transaction message size from
172growing too large. In this case, the segment information in the Statement
173submessages is not helpful if the Statement isn't segmented. We need this
174information in the Transaction message itself.
175
176Segmenting a Single SQL Statement
177*********************************
178
179When a regular SQL statement modifies or inserts more rows than a
180certain threshold, Drizzle's replication services component will begin
181sending Transaction messages to replicas which contain a chunk
182(or "segment") of the data which has been changed on the publisher.
183
184When data is inserted, updated, or modified in the database, a
185header containing information about modified tables and fields is
186matched with one or more data segments which contain the actual
187values changed in the statement.
188
189It's easiest to understand this mechanism by following through a real-world
190scenario.
191
192Suppose the following table:
193
194.. code-block:: mysql
195
196 CREATE TABLE test.person
197 (
198 id INT NOT NULL AUTO_INCREMENT PRIMARY KEY
199 , first_name VARCHAR(50)
200 , last_name VARCHAR(50)
201 , is_active CHAR(1) NOT NULL DEFAULT 'Y'
202 );
203
204Also suppose that test.t1 contains 1 million records.
205
206Next, suppose a client issues the SQL statement:
207
208.. code-block:: mysql
209
210 UPDATE test.person SET is_active = 'N';
211
212It is clear that one million records could be updated by this statement
213(we say, "could be" since Drizzle does not actually update a record if
214the UPDATE would not change the existing record...).
215
216In order to prevent the publishing server from having to construct an
217enormous Transaction message, Drizzle's replication services component
218will do the following:
219
220#. Construct a Transaction message with a transaction context containing
221 information about the originating server, the transaction ID, and
222 timestamp information.
223#. Construct an UpdateHeader message with information about the tables
224 and fields involved in the UPDATE statement. Push this UpdateHeader
225 message onto the Transaction message's statement vector.
226#. Construct an UpdateData message. Set the *segment_id* member to 1.
227 Set the *end_segment* member to true.
228#. For every record updated in a storage engine, the ReplicationServices
229 component builds a new UpdateRecord message and appends this message
230 to the aforementioned UpdateData message's record vector.
231#. After a certain threshold of records is reached, the
232 ReplicationServices component sets the current UpdateData message's
233 *end_segment* member to false, and proceeds to send the Transaction
234 message to replicators.
235#. The ReplicationServices component then constructs a new Transaction
236 message and constructs a transaction context with the same
237 transaction ID and server information.
238#. A new UpdateData message is created. The message's *segment_id* is
239 set to N+1 and as new records are updated, new UpdateRecord messages
240 are appended to the UpdateData message's record vector.
241#. While records are being updated, we repeat steps 5 through 7, with
242 only the final UpdateData message having its *end_segment* member set
243 to true.
244
245Segmenting a Transaction
246************************
247
248The Transaction protobuf message also contains *segment_id* member and a
249*end_segment* member. These values are also set appropriately when a
250Statement sub-message is segmented, as described above.
251
252These values are also set when a Transaction must be segmented along
253individual Statement boundaries (i.e., the Statement message itself
254is **not** segmented). In either case, it is enough to check the
255*end_segment* and *segment_id* values of the Transaction message
256to determine if this is a multi-message transaction.
257
258Handling ROLLBACKs
259##################
260
261Both transactions and individual statements may be rolled back.
262
263When a transaction is rolled back, one of two things happen depending
264on whether the transaction is made up of either a single Transaction
265message, or if it is made up of multiple Transaction messages (e.g, bulk
266load).
267
268* For a transaction encapsulated entirely within a single Transaction
269 message, the entire message is simply discarded and not sent through
270 the replication stream.
271* For a transaction which is made up of multiple messages, and at least
272 one message has already been sent through the replication stream, then
273 the Transaction message will contain a Statement message with type =
274 ROLLBACK. This signifies to rollback the entire transaction.
275
276A special Statement message type, ROLLBACK_STATEMENT, is used when
277we have a segmented Statement message (see above) and we need to tell the
278receiver to undo any changes made for this single statement, but not
279for the entire transaction. If the receiver cannot handle rolling back
280a single statement, then a message buffering strategy should be employed
281to guarantee that a statement was indeed applied successfully before
282executing on the receiver.
283
284.. _replication_streams:
285
286Replication Streams
287###################
288
289The Drizzle kernel handles delivering replication messages to plugins by
290maintaining a list of replication streams. A stream is represented as a
291registered *replicator* and *applier* pair.
292
293When a replication message is generated within the kernel, the replication
294services module of the kernel will send this message to each registered
295*replicator*. The *replicator* will then do something useful with it and
296send it to each *applier* with which it is associated.
297
298Replicators
299***********
300
301A registered *replicator* is a plugin that implements the TransactionReplicator
302API. Each replicator will be plugged into the kernel to receive the Google
303Protobuf messages that are generated as the database is changed. Ideally,
304each registered replicator will transform or modify the messages it receives
305to implement a specific behavior. For example, filtering by schema name.
306
307Each registered replicator should have a unique name. The default replicator,
308cleverly named **default_replicator**, does no transformation at all on the
309replication messages.
310
311Appliers
312********
313
314A registered *applier* is a plugin that implements the TransactionApplier
315API. Appliers are responsible for applying the replication messages that it
316will receive from a registered replicator. The word "apply" is used loosely
317here. An applier may do anything with the replication messages that provides
318useful behavior. For example, an applier may simply write the messages to a
319file on disk, or it may send the messages over the network to some other
320service to be processed.
321
322At the point of registration with the Drizzle kernel, each applier specifies
323the name of a registered replicator that it should be attached to in order to
324make the replication stream pair.
3250
=== added directory 'docs/replication/examples'
=== added file 'docs/replication/examples/index.rst'
--- docs/replication/examples/index.rst 1970-01-01 00:00:00 +0000
+++ docs/replication/examples/index.rst 2012-03-18 00:36:19 +0000
@@ -0,0 +1,32 @@
1.. _replication_examples:
2
3Examples
4========
5
6These replication examples require a working knowledge of:
7
8* :ref:`configuring_drizzle`
9* :ref:`administering_drizzle`
10* :ref:`drizzle_replication`
11
12The examples are complete, but they may not explain every step in detail, and they probably need to be adapted to other environments. In other words: the examples are guides, not drop-in solutions. Free and commercial :ref:`help` is available.
13
14.. _slave_examples:
15
16Slave Applier
17-------------
18
19.. toctree::
20 :maxdepth: 1
21 :glob:
22
23 slave/*
24
25RabbitMQ Applier
26----------------
27
28.. toctree::
29 :maxdepth: 1
30 :glob:
31
32 rabbitmq/*
033
=== added directory 'docs/replication/examples/rabbitmq'
=== added file 'docs/replication/examples/rabbitmq/basic.rst'
--- docs/replication/examples/rabbitmq/basic.rst 1970-01-01 00:00:00 +0000
+++ docs/replication/examples/rabbitmq/basic.rst 2012-03-18 00:36:19 +0000
@@ -0,0 +1,55 @@
1.. _basic_rabbitmq_example:
2
3Basic RabbitMQ
4==============
5
6:Synopsis: Set up replication to a RabbitMQ server
7:Replicator: :ref:`default_replicator`
8:Applier: :ref:`rabbitmq_applier`
9:Version: :ref:`rabbitmq_0.1_drizzle_7.0`
10:Authors: Marcus Eriksson
11
12First install a recent version of RabbitMQ, then install librabbitmq, the C library for talking to the RabbitMQ server:
13
14.. code-block:: bash
15
16 $ hg clone http://hg.rabbitmq.com/rabbitmq-codegen/
17 $ hg clone http://hg.rabbitmq.com/rabbitmq-c/
18 $ cd rabbitmq-c
19 $ autoreconf -f -i
20 $ ./configure
21 $ make
22 $ make install
23
24Now you probably need to rebuild Drizzle since the :program:`rabbitmq` plugin is not built if librabbitmq is not installed.
25
26Finally, start :program:`drizzled` like:
27
28.. code-block:: bash
29
30 sbin/drizzled \
31 --daemon \
32 --pid-file /var/run/drizzled.pid \
33 --plugin-add rabbitmq \
34 >> /var/log/drizzled.log 2>&1
35
36To verify that it works, you can start a generic rabbitmq listener from librabbitmq:
37
38.. code-block:: bash
39
40 $ amqp_listen localhost 5672 ReplicationExchange ReplicationRoutingKey
41
42And you should see something like this when you do an INSERT/CREATE/.. (just not a select) in your newly built Drizzle instance::
43
44 Result 0
45 Frame type 1, channel 1
46 Method AMQP_BASIC_DELIVER_METHOD
47 Delivery 1, exchange ReplicationExchange routingkey ReplicationRoutingKey
48
49 00000000: 0A 17 08 01 10 87 36 18 : F0 FA D9 99 FA F1 A7 02 ......6.........
50 00000010: 20 99 81 DA 99 FA F1 A7 : 02 12 40 08 01 10 F2 FA .........@.....
51 00000020: D9 99 FA F1 A7 02 18 FC : FA D9 99 FA F1 A7 02 2A ...............*
52 00000030: 17 0A 06 0A 01 62 12 01 : 61 12 06 08 04 12 02 69 .....b..a......i
53 00000040: 64 12 05 08 01 12 01 74 : 32 11 08 01 10 01 1A 0B d......t2.......
54 00000050: 0A 01 32 0A 02 61 61 10 : 00 10 00 20 01 28 01 ..2..aa.... .(.
55 0000005F:
056
=== added directory 'docs/replication/examples/slave'
=== added file 'docs/replication/examples/slave/provision_new_slave.rst'
--- docs/replication/examples/slave/provision_new_slave.rst 1970-01-01 00:00:00 +0000
+++ docs/replication/examples/slave/provision_new_slave.rst 2012-03-18 00:36:19 +0000
@@ -0,0 +1,61 @@
1.. _provision_new_slave_example:
2
3Provision New Slave
4===================
5
6:Synopsis: Provision a new slave from a backup of a master
7:Replicator: :ref:`default_replicator`
8:Applier: :ref:`slave`
9:Version: :ref:`slave_1.1_drizzle_7.1`
10:Authors: Marisa Plumb, Kent Bozlinski, Daniel Nichter
11
12The basic formula for creating a new slave for an existing master is:
13
14#. Make a backup of the master databases.
15#. Record the max committed transaction ID of the master at the point the backup was made.
16#. Restore the backup on the new slave.
17#. Start the new slave from the recorded max committed transaction ID of the master.
18
19.. program:: drizzledump
20
21Steps #1 and #2 are performed using the :ref:`drizzledump` and its :option:`--single-transaction` option which prints a comment near the beginning of the dump output with the InnoDB transaction log metadata:
22
23.. code-block:: bash
24
25 master$ drizzledump --all-databases --single-transaction > master-backup.sql
26
27 master$ grep SYS_REPLICATION_LOG master-backup.sql
28 -- SYS_REPLICATION_LOG: COMMIT_ID = 3303, ID = 3500
29
30The ``SYS_REPLICATION_LOG`` comment provides the replication log metadata needed to start a new slave. There are two values:
31
32COMMIT_ID
33 The commit sequence number recorded for the most recently executed transaction stored in the transaction log. This value is used to determine proper commit order within the log. ``ID``, the unique transaction identifier, cannot be used because it is assigned when the transaction is started, not when it is committed.
34
35ID
36 The unique transaction identifier associated with the most recently executed transaction stored in the transaction log.
37
38For step #3, start the slave *without* the :ref:`slave_plugin` to prevent it from reading from the master until the backup is imported. Then import the backup on the slave:
39
40.. code-block:: bash
41
42 slave$ drizzle < master-backup.sql
43
44Stop the slave once the backup finishes importing.
45
46For step #4, add the :ref:`max-commit-id option <slave_cfg_master_options>` to the :ref:`slave_config_file`:
47
48.. code-block:: ini
49
50 # Example of existing lines
51 [master1]
52 master-host=10.0.0.1
53 master-user=slave
54 master-pass=foo
55
56 # Add this line
57 max-commit-id=3303
58
59The value for :ref:`max-commit-id <slave_cfg_master_options>` is the ``COMMIT_ID`` value from the ``SYS_REPLICATION_LOG`` comment in the master dump file (steps #1 and #2). This value defines the committed transaction ID on the master from which the slave will start applying transactions.
60
61Finally, start the slave *with* the :ref:`slave_plugin` and verify that replication is working (see :ref:`slave_admin`).
062
=== added file 'docs/replication/examples/slave/simple_master_slave.rst'
--- docs/replication/examples/slave/simple_master_slave.rst 1970-01-01 00:00:00 +0000
+++ docs/replication/examples/slave/simple_master_slave.rst 2012-03-18 00:36:19 +0000
@@ -0,0 +1,172 @@
1.. program:: drizzled
2
3.. _simple_master_slave_example:
4
5Simple Master-Slave
6===================
7
8:Synopsis: Set up one master and one slave, neither of which have existing data
9:Replicator: :ref:`default_replicator`
10:Applier: :ref:`slave`
11:Version: :ref:`slave_1.1_drizzle_7.1`
12:Authors: Daniel Nichter
13
14This example uses new servers that are not yet configured and do not yet have any data. If Drizzle was installed from a binary package or repository, then it may already be configured, in which case the following steps may not work or conflict with the existing configuration. In any case, the principles in this example can be adapted to different configurations.
15
16Master Setup
17------------
18
19First, temporarily start the master server with no :ref:`authentication` in order to create the user accounts.
20
21.. code-block:: bash
22
23 sbin/drizzled >> /var/log/drizzled.log 2>&1 &
24
25Once Drizzle is running, create the user accounts:
26
27.. code-block:: mysql
28
29 drizzle> CREATE SCHEMA auth;
30
31 drizzle> USE auth;
32
33 drizzle> CREATE TABLE users (
34 -> user VARCHAR(255) NOT NULL,
35 -> password VARCHAR(40),
36 -> UNIQUE INDEX user_idx (user)
37 -> );
38
39 drizzle> INSERT INTO auth.users (user, password)
40 -> VALUES ('root', MYSQL_PASSWORD('foo')),
41 -> ('slave', MYSQL_PASSWORD('foo'));
42
43 drizzle> shutdown;
44
45Once Drizzle has stopped running, restart it with the :ref:`auth_schema_plugin` plugin and other plugins required for :ref:`configuring_a_master`:
46
47.. code-block:: bash
48
49 sbin/drizzled \
50 --daemon \
51 --pid-file /var/run/drizzled/drizzled.pid \
52 --plugin-remove auth_all \
53 --plugin-add auth_schema \
54 --innodb.replication-log \
55 >> /var/log/drizzled.log 2>&1
56
57See the options used to start Drizzle for more information:
58
59* :option:`--daemon`
60* :option:`--pid-file`
61* :option:`--plugin-add`
62* :option:`--innodb.replication-log`
63
64Other options and :ref:`plugins` can be used if necessary.
65
66Verify that the master is running and writing :ref:`replication_events` to the :ref:`innodb_transaction_log`:
67
68.. code-block:: mysql
69
70 $ drizzle --user=root --password=foo
71
72 drizzle> SELECT ID FROM DATA_DICTIONARY.SYS_REPLICATION_LOG LIMIT 1;
73 +------+
74 | ID |
75 +------+
76 | 772 |
77 +------+
78
79The query should return a row. This is the table from which slaves read replication events.
80
81The master is now ready to replicate to slaves.
82
83Slave Setup
84-----------
85
86The slave must be on a different sever than the master, else the two servers will have port conflicts.
87
88Since the slave is new and has no data, it will replicate all data from the master, including the :ref:`auth_schema_plugin` table just created.
89
90First, write a :ref:`slave_config_file`:
91
92.. code-block:: ini
93
94 [master1]
95 master-host=10.0.0.1
96 master-user=slave
97 master-pass=foo
98
99The :ref:`master-host option <slave_cfg_master_options>` must be set to the master server's address. 10.0.0.1 is just an example.
100
101Save the config file in :file:`/etc/drizzle/slave.cfg`.
102
103Then start the slave with the plugins required for :ref:`configuring_a_slave`:
104
105.. code-block:: bash
106
107 sbin/drizzled \
108 --daemon \
109 --pid-file /var/run/drizzled/drizzled.pid \
110 --plugin-add slave \
111 --slave.config-file /etc/drizzle/slave.cfg \
112 >> /var/log/drizzled.log 2>&1
113
114Verify that the slave is running and applying replication events from the master:
115
116.. code-block:: mysql
117
118 $ drizzle --user=root --password=foo
119
120 drizzle> SELECT * FROM sys_replication.io_state\G
121 *************************** 1. row ***************************
122 master_id: 1
123 status: RUNNING
124 error_msg:
125
126 drizzle> SELECT * FROM sys_replication.applier_state\G
127 *************************** 1. row ***************************
128 master_id: 1
129 last_applied_commit_id: 23
130 originating_server_uuid: 98ECEA09-BA65-489D-9382-F8D15098B1AE
131 originating_commit_id: 23
132 status: RUNNING
133 error_msg:
134
135The column values will vary, but the important column is ``status`` :ref:`RUNNING <slave_thread_statuses>` for both tables.
136
137After at least :ref:`applier-thread-sleep <slave_cfg_common_options>` or :ref:`io-thread-sleep <slave_cfg_common_options>` seconds, the :ref:`auth_schema_plugin` table should replicate from the master:
138
139.. code-block:: mysql
140
141 drizzle> SELECT * FROM auth.users;
142 +-------+------------------------------------------+
143 | user | password |
144 +-------+------------------------------------------+
145 | root | F3A2A51A9B0F2BE2468926B4132313728C250DBF |
146 | slave | F3A2A51A9B0F2BE2468926B4132313728C250DBF |
147 +-------+------------------------------------------+
148
149If not, then check the error log (:file:`/var/log/drizzled.log`) for errors.
150
151Check the :ref:`slave_lag` to ensure that the slave has applied all committed transactions from the master.
152
153Once the slave is fully caught up to the master, stop it, then start it again with the :ref:`auth_schema_plugin` plugin:
154
155.. code-block:: bash
156
157 sbin/drizzled \
158 --daemon \
159 --pid-file /var/run/drizzled/drizzled.pid \
160 --plugin-add slave \
161 --slave.config-file /etc/drizzle/slave.cfg \
162 --plugin-remove auth_all \
163 --plugin-add auth_schema \
164 >> /var/log/drizzled.log 2>&1
165
166The root username and password should be required:
167
168.. code-block:: bash
169
170 $ drizzle --user=root --password=foo
171
172The :ref:`sys_replication_tables` should still show that the :ref:`slave_threads` are :ref:`RUNNING <slave_thread_statuses>`. Any changes on the master should replicate to the slave within a few seconds. If any problems occur, consult the :ref:`slave` documentation or ask for :ref:`help`. Else, congratulations: you are up and running with slave-based Drizzle replication! Be sure to familiarize yourself with :ref:`slave_admin`.
0173
=== added file 'docs/replication/index.rst'
--- docs/replication/index.rst 1970-01-01 00:00:00 +0000
+++ docs/replication/index.rst 2012-03-18 00:36:19 +0000
@@ -0,0 +1,172 @@
1.. program:: drizzled
2
3.. _drizzle_replication:
4
5Drizzle Replication
6===================
7
8.. topic:: Want to skip the details and setup Drizzle replicaiton as quickly as possible?
9
10 Then start with the
11 :ref:`Simple Master-Slave example <simple_master_slave_example>`.
12 Otherwise, start here and read each section in sequence for a complete
13 understanding of Drizzle replication.
14
15Drizzle encodes replication events as
16`Google Protocol Buffer <http://code.google.com/p/protobuf/>`_ (GPB) messages
17and transmits them asynchronously through replication streams. A replication
18stream is a pair of one replicator and one applier. The kernel encodes
19replication events, sends them to all available replicators, which in turn send
20them to their paired appliers. For Drizzle, the replication process ends at
21this point, but appliers are responsible for applying replication events to a
22service: another Drizzle server (a slave), a message queue, etc.
23
24.. code-block:: none
25
26 Drizzle External
27 ========================================================== ========
28 User Kernel Plugins
29 ===== ================= =========================
30 Replication Streams
31 =========================
32 event --> encode GBP(event) +--> replicator1 +--> applier1 --> service1
33 | |
34 | +--> applier2 --> service2
35 |
36 +--> replicator2 +--> applier3 --> service3
37
38Replicators and appliers are implemented by plugins, so Drizzle replication
39is extensible and varies depending on which plugins are used. Drizzle
40includes plugins to implement several replication systems:
41
42* :ref:`Master-slave replication, including multi-master <slave_applier>`
43* :ref:`Replication to a RabbitMQ server <rabbitmq_applier>`
44* :ref:`Replication to a ZeroMQ socket <zeromq_applier>`
45
46Master-slave is the most common replication system; it resembles other
47database servers like MySQL.
48
49Unlike other database servers, the Drizzle kernel plays only a minimal role
50in the replication process: it simply encodes and sends replication events
51to available replicators. Consequently, there are very few
52:ref:`drizzled replication options <drizzled_replication_options>`, and
53Drizzle replication is primarily configured by replicator and applier
54plugin options.
55
56In summary, Drizzle replication:
57
58* is asynchronous
59* encodes replication events as Google Buffer Protocol messages
60* sends replication events through replication streams (unique replicator-applier pairs)
61* only uses the kernel to encode and send replication events to avaiable replicators
62* is primarily implemented and configured by replicator and applier plugins
63* is extensible
64
65Learned enough? Ready to start using Drizzle replication? Then jump to the
66:ref:`replication examples <replication_examples>`. Otherwise, continue
67reading for more details about all the parts and processes of Drizzle
68replication.
69
70.. _replication_events:
71
72Replication Events
73------------------
74
75Replication events are, in general, any SQL statements which change data or
76schema objects on a server. :doc:`/dml` queries are the primary cause of
77replication events, but other statements like setting global variables or
78altering schemas or tables may also cause replication events.
79
80The Drizzle kernel sends every replication event to every applier (that is
81to say, every loaded applier plugin), but appliers determine if replicaiton
82events are sent to their paired appliers. For example, a replicator may
83filter certain replication events. Likewise, appliers determine if (and how)
84replication events are ultimately applied.
85
86Replication events are not logged or saved by the Drizzle kernel or any
87replicators that ship with Drizzle. An applier may log replication events
88that it receives from its replicator.
89
90Every replication event is encapsulated as an atomic transaction, including
91bulk and mutli-statement events.
92
93Drizzle relinquishes all control of replication events once they enter a
94replication stream. Replicators and appliers are responsbile for handling
95replication events correctly and efficiently.
96
97.. _replication_streams:
98
99Replication Streams
100-------------------
101
102Replication stream are logical conduits created by pairing one replicator
103with one applier. As logical entities, replicaiton streams exist only inside
104the :program:`drizzled` process and cannot be accessed externally. However,
105some appliers create or access ports or sockets which allows indirect access
106to the replication stream. Since replicators and appliers are implemented
107by plugins, one could in theory program a custom applier or replicator to
108provide a socket or port for direct access into the replication stream.
109
110When :program:`drizlzed` starts, it creates replication streams automatically
111based on which replicators are loaded and which appliers are loaded and
112configured to use them. For example, an applier plugin may be configured
113to use a specific replicator, in which case :program:`drizzled` pairs the
114applier to the specified replicator. The user does not need to perform
115special steps to create a replication stream.
116
117Replication stream cannot be dynamically recreated; the user must stop
118Drizzle, reconfigure the replicator or applier, and then restart Drizzle to
119let it automatically recreate the new replication stream.
120
121.. _originating_server:
122
123Originating Server
124------------------
125
126The originating server of a replication event is the server on which the
127SQL statement that caused the replication was first executed. Since one
128replicaiton event may be applied to several services (by passing through
129multiple replication streams), the originating server uniquely identifies
130the true origin of a replication event versus its most immediate upstream
131origin which may have received the replication event from any number of
132additional upstream sources.
133
134Drizzle automatically generates a UUID for every server, saved in the
135:file:`server.uuid` file in the :option:`--datadir` directory. This UUID
136is included with every replication event that originates from the server.
137
138An originating server may or may not contain both end points of a replication
139stream. Replicators are always local to (loaded and ran from) the originating
140server from which they receive replication events, but appliers may be local
141or remote (loaded and ran on a different server). The external service to
142which the applier applies replication events is usually another server,
143not the originating server, but an applier could, in theory, apply events
144from and to the same originating server.
145
146Configuration
147-------------
148
149Drizzle replication is primarily configured by options specific to
150each :ref:`replicator <replicators>` and :ref:`applier <appliers>`.
151
152The Drizzle kernel has very few :ref:`drizzled_replication_options` which
153typically do not need to be changed:
154
155:option:`--transaction-message-threshold`
156 Controls the size, in bytes, of the transaction messages.
157 When a transaction message exceeds this size, a new transaction message
158 with the same transaction ID will be created to continue the replication
159 events. See :ref:`bulk-operations`.
160
161:option:`--replicate-query`
162 Controls whether the originating SQL query will be included within each
163 statement message contained in the enclosing transaction message. The
164 default global value is FALSE which will not include the query in the
165 messages. It can be controlled per session, as well. For example:
166
167 .. code-block:: mysql
168
169 drizzle> SET @@replicate_query = 1;
170
171 The stored query should be used as a guide only, and never executed
172 on a slave to perform replication as this will lead to incorrect results.
0173
=== added directory 'docs/replication/messages'
=== added file 'docs/replication/messages/index.rst'
--- docs/replication/messages/index.rst 1970-01-01 00:00:00 +0000
+++ docs/replication/messages/index.rst 2012-03-18 00:36:19 +0000
@@ -0,0 +1,251 @@
1Messages
2********
3
4Message Definitions
5===================
6
7The GPB messages are defined in .proto files in the drizzled/message
8directory of the Drizzle source code. The primary definition file is
9transaction.proto. Messages defined in this file are related in the
10following ways::
11
12
13 ------------------------------------------------------------------
14 | |
15 | Transaction message |
16 | |
17 | ----------------------------------------------------------- |
18 | | | |
19 | | TransactionContext message | |
20 | | | |
21 | ----------------------------------------------------------- |
22 | ----------------------------------------------------------- |
23 | | | |
24 | | Statement message 1 | |
25 | | | |
26 | ----------------------------------------------------------- |
27 | ----------------------------------------------------------- |
28 | | | |
29 | | Statement message 2 | |
30 | | | |
31 | ----------------------------------------------------------- |
32 | ... |
33 | ----------------------------------------------------------- |
34 | | | |
35 | | Statement message N | |
36 | | | |
37 | ----------------------------------------------------------- |
38 ------------------------------------------------------------------
39
40with each Statement message looking like so::
41
42 ------------------------------------------------------------------
43 | |
44 | Statement message |
45 | |
46 | ----------------------------------------------------------- |
47 | | | |
48 | | Common information | |
49 | | | |
50 | | - Type of Statement (INSERT, DELETE, etc) | |
51 | | - Start Timestamp | |
52 | | - End Timestamp | |
53 | | - (OPTIONAL) Actual SQL query string | |
54 | | | |
55 | ----------------------------------------------------------- |
56 | ----------------------------------------------------------- |
57 | | | |
58 | | Statement subclass message 1 (see below) | |
59 | | | |
60 | ----------------------------------------------------------- |
61 | ... |
62 | ----------------------------------------------------------- |
63 | | | |
64 | | Statement subclass message N (see below) | |
65 | | | |
66 | ----------------------------------------------------------- |
67 ------------------------------------------------------------------
68
69The Transaction Message
70=======================
71
72The main "envelope" message which represents an atomic transaction
73which changed the state of a server is the Transaction message class.
74
75The Transaction message contains two pieces:
76
77#. A TransactionContext message containing information about the
78 transaction as a whole, such as the ID of the executing server,
79 the start and end timestamp of the transaction, segmenting
80 metadata and a unique identifier for the transaction.
81#. A vector of Statement messages representing the distinct SQL
82 statements which modified the state of the server. The Statement
83 message is, itself, a generic envelope message containing a
84 sub-message which describes the specific data modification which
85 occurred on the server (such as, for instance, an INSERT statement).
86
87The Statement Message
88=====================
89
90The generic "envelope" message containing information common to each
91SQL statement executed against a server (such as a start and end timestamp
92and the type of the SQL statement) as well as a Statement subclass message
93describing the specific data modification event on the server.
94
95Each Statement message contains a type member which indicates how readers
96of the Statement should construct the inner Statement subclass representing
97a data change.
98
99Statements are recorded separately as sometimes individual statements
100have to be rolled back.
101
102
103.. _bulk-operations:
104
105How Bulk Operations Work
106========================
107
108Certain operations which change large volumes of data on a server
109present a specific set of problems for a transaction coordinator or
110replication service. If all operations must complete atomically on a
111publishing server before replicas are delivered the complete
112transactional unit:
113
114#. The publishing server could consume a large amount of memory
115 building an in-memory Transaction message containing all the
116 operations contained in the entire transaction.
117#. A replica, or subscribing server, is wasting time waiting on the
118 eventual completion (commit) of the large transaction on the
119 publishing server. It could be applying pieces of the large
120 transaction in the meantime...
121
122In order to prevent the problems inherent in (1) and (2) above, Drizzle's
123replication system uses a mechanism which provides bulk change
124operations.
125
126A single transaction in the database can possibly be represented with
127multiple protobuf Transaction messages if the message grows too large.
128This can happen if you have a bulk transaction, or a single statement
129affecting a very large number of rows, or just a large transaction with
130many statements/changes.
131
132For the first two examples, it is likely that the Statement sub-message
133itself will get segmented, causing another Transaction message to be
134created to hold the rest of the Statement's row changes. In these cases,
135it is enough to look at the segment information stored in the Statement
136message (see example below).
137
138For the last example, the Statement sub-messages may or may not be
139segmented, but we could still need to split the individual Statements up into
140multiple Transaction messages to keep the Transaction message size from
141growing too large. In this case, the segment information in the Statement
142submessages is not helpful if the Statement isn't segmented. We need this
143information in the Transaction message itself.
144
145Segmenting a Single SQL Statement
146=================================
147
148When a regular SQL statement modifies or inserts more rows than a
149certain threshold, Drizzle's replication services component will begin
150sending Transaction messages to replicas which contain a chunk
151(or "segment") of the data which has been changed on the publisher.
152
153When data is inserted, updated, or modified in the database, a
154header containing information about modified tables and fields is
155matched with one or more data segments which contain the actual
156values changed in the statement.
157
158It's easiest to understand this mechanism by following through a real-world
159scenario.
160
161Suppose the following table:
162
163.. code-block:: mysql
164
165 CREATE TABLE test.person
166 (
167 id INT NOT NULL AUTO_INCREMENT PRIMARY KEY
168 , first_name VARCHAR(50)
169 , last_name VARCHAR(50)
170 , is_active CHAR(1) NOT NULL DEFAULT 'Y'
171 );
172
173Also suppose that test.t1 contains 1 million records.
174
175Next, suppose a client issues the SQL statement:
176
177.. code-block:: mysql
178
179 UPDATE test.person SET is_active = 'N';
180
181It is clear that one million records could be updated by this statement
182(we say, "could be" since Drizzle does not actually update a record if
183the UPDATE would not change the existing record...).
184
185In order to prevent the publishing server from having to construct an
186enormous Transaction message, Drizzle's replication services component
187will do the following:
188
189#. Construct a Transaction message with a transaction context containing
190 information about the originating server, the transaction ID, and
191 timestamp information.
192#. Construct an UpdateHeader message with information about the tables
193 and fields involved in the UPDATE statement. Push this UpdateHeader
194 message onto the Transaction message's statement vector.
195#. Construct an UpdateData message. Set the *segment_id* member to 1.
196 Set the *end_segment* member to true.
197#. For every record updated in a storage engine, the ReplicationServices
198 component builds a new UpdateRecord message and appends this message
199 to the aforementioned UpdateData message's record vector.
200#. After a certain threshold of records is reached, the
201 ReplicationServices component sets the current UpdateData message's
202 *end_segment* member to false, and proceeds to send the Transaction
203 message to replicators.
204#. The ReplicationServices component then constructs a new Transaction
205 message and constructs a transaction context with the same
206 transaction ID and server information.
207#. A new UpdateData message is created. The message's *segment_id* is
208 set to N+1 and as new records are updated, new UpdateRecord messages
209 are appended to the UpdateData message's record vector.
210#. While records are being updated, we repeat steps 5 through 7, with
211 only the final UpdateData message having its *end_segment* member set
212 to true.
213
214Segmenting a Transaction
215========================
216
217The Transaction protobuf message also contains *segment_id* member and a
218*end_segment* member. These values are also set appropriately when a
219Statement sub-message is segmented, as described above.
220
221These values are also set when a Transaction must be segmented along
222individual Statement boundaries (i.e., the Statement message itself
223is **not** segmented). In either case, it is enough to check the
224*end_segment* and *segment_id* values of the Transaction message
225to determine if this is a multi-message transaction.
226
227Handling ROLLBACKs
228==================
229
230Both transactions and individual statements may be rolled back.
231
232When a transaction is rolled back, one of two things happen depending
233on whether the transaction is made up of either a single Transaction
234message, or if it is made up of multiple Transaction messages (e.g, bulk
235load).
236
237* For a transaction encapsulated entirely within a single Transaction
238 message, the entire message is simply discarded and not sent through
239 the replication stream.
240* For a transaction which is made up of multiple messages, and at least
241 one message has already been sent through the replication stream, then
242 the Transaction message will contain a Statement message with type =
243 ROLLBACK. This signifies to rollback the entire transaction.
244
245A special Statement message type, ROLLBACK_STATEMENT, is used when
246we have a segmented Statement message (see above) and we need to tell the
247receiver to undo any changes made for this single statement, but not
248for the entire transaction. If the receiver cannot handle rolling back
249a single statement, then a message buffering strategy should be employed
250to guarantee that a statement was indeed applied successfully before
251executing on the receiver.
0252
=== added directory 'docs/replication/replicators'
=== added file 'docs/replication/replicators/default.rst'
--- docs/replication/replicators/default.rst 1970-01-01 00:00:00 +0000
+++ docs/replication/replicators/default.rst 2012-03-18 00:36:19 +0000
@@ -0,0 +1,64 @@
1.. _default_replicator:
2
3Default Replicator
4==================
5
6The default replicator plugin, cleverly named ``default_replicator``,
7does not modify or filter any replication events; it simply sends every
8replication event it receives from the Drizzle kernel to every applier
9with which it is paired.
10
11:ref:`appliers` default to this replicator.
12
13.. _default_replicator_loading:
14
15Loading
16-------
17
18This plugin is loaded by default. To stop the plugin from loading by
19default, start :program:`drizzled` with::
20
21 --plugin-remove=default_replicator
22
23.. seealso:: :ref:`drizzled_plugin_options` for more information about adding and removing plugins.
24
25.. _default_replicator_configuration:
26
27Configuration
28-------------
29
30This plugin does not have any command line options.
31
32.. _default_replicator_variables:
33
34Variables
35---------
36
37This plugin does not register any variables.
38
39.. _default_replicator_authors:
40
41Authors
42-------
43
44Jay Pipes
45
46.. _default_replicator_version:
47
48Version
49-------
50
51This documentation applies to **default_replicator 1.0**.
52
53To see which version of the plugin a Drizzle server is running, execute:
54
55.. code-block:: mysql
56
57 SELECT MODULE_VERSION FROM DATA_DICTIONARY.MODULES WHERE MODULE_NAME='default_replicator'
58
59Changelog
60---------
61
62v1.0
63^^^^
64* First release.
065
=== added file 'docs/replication/replicators/filtered.rst'
--- docs/replication/replicators/filtered.rst 1970-01-01 00:00:00 +0000
+++ docs/replication/replicators/filtered.rst 2012-03-18 00:36:19 +0000
@@ -0,0 +1,143 @@
1.. _filtered_replicator:
2
3Filtered Replicator
4===================
5
6The filtered replicator plugin, named ``filtered_replicator``, filters
7replication events based on schema name or table name. Regular expressions
8can be used for the schema and table names.
9
10.. _filtered_replicator_loading:
11
12Loading
13-------
14
15To load this plugin, start :program:`drizzled` with::
16
17 --plugin-add=filtered_replicator
18
19Loading the plugin may not enable or configure it. See the plugin's
20:ref:`filtered_replicator_configuration` and :ref:`filtered_replicator_variables`.
21
22.. seealso:: :ref:`drizzled_plugin_options` for more information about adding and removing plugins.
23
24.. _filtered_replicator_configuration:
25
26Configuration
27-------------
28
29These command line options configure the plugin when :program:`drizzled`
30is started. See :ref:`command_line_options` for more information about
31specifying command line options.
32
33.. program:: drizzled
34
35.. option:: --filtered-replicator.filteredschemas ARG
36
37 :Default:
38 :Variable: :ref:`filtered_replicator_filteredschemas <filtered_replicator_filteredschemas>`
39
40 Comma-separated list of schemas to exclude from replication.
41
42.. option:: --filtered-replicator.filteredtables ARG
43
44 :Default:
45 :Variable: :ref:`filtered_replicator_filteredtables <filtered_replicator_filteredtables>`
46
47 Comma-separated list of tables to exclude from replication.
48
49.. option:: --filtered-replicator.schemaregex ARG
50
51 :Default:
52 :Variable: :ref:`filtered_replicator_schemaregex <filtered_replicator_schemaregex>`
53
54 Regular expression to apply to schemas to exclude from replication.
55
56.. option:: --filtered-replicator.tableregex ARG
57
58 :Default:
59 :Variable: :ref:`filtered_replicator_tableregex <filtered_replicator_tableregex>`
60
61 Regular expression to apply to tables to exclude from replication.
62
63.. _filtered_replicator_variables:
64
65Variables
66---------
67
68These variables show the running configuration of the plugin.
69See `variables` for more information about querying and setting variables.
70
71.. _filtered_replicator_filteredschemas:
72
73* ``filtered_replicator_filteredschemas``
74
75 :Scope: Global
76 :Dynamic: No
77 :Option: :option:`--filtered-replicator.filteredschemas`
78
79 Comma-separated list of schemas to exclude from replication.
80
81.. _filtered_replicator_filteredtables:
82
83* ``filtered_replicator_filteredtables``
84
85 :Scope: Global
86 :Dynamic: No
87 :Option: :option:`--filtered-replicator.filteredtables`
88
89 Comma-separated list of tables to exclude from replication.
90
91.. _filtered_replicator_schemaregex:
92
93* ``filtered_replicator_schemaregex``
94
95 :Scope: Global
96 :Dynamic: No
97 :Option: :option:`--filtered-replicator.schemaregex`
98
99 Regular expression to apply to schemas to exclude from replication.
100
101.. _filtered_replicator_tableregex:
102
103* ``filtered_replicator_tableregex``
104
105 :Scope: Global
106 :Dynamic: No
107 :Option: :option:`--filtered-replicator.tableregex`
108
109 Regular expression to apply to tables to exclude from replication.
110
111.. _filtered_replicator_examples:
112
113Examples
114--------
115
116Sorry, there are no examples for this plugin.
117
118.. _filtered_replicator_authors:
119
120Authors
121-------
122
123Padraig O Sullivan
124
125.. _filtered_replicator_version:
126
127Version
128-------
129
130This documentation applies to **filtered_replicator 0.2**.
131
132To see which version of the plugin a Drizzle server is running, execute:
133
134.. code-block:: mysql
135
136 SELECT MODULE_VERSION FROM DATA_DICTIONARY.MODULES WHERE MODULE_NAME='filtered_replicator'
137
138Chagnelog
139---------
140
141v0.2
142^^^^
143* First release.
0144
=== added file 'docs/replication/replicators/index.rst'
--- docs/replication/replicators/index.rst 1970-01-01 00:00:00 +0000
+++ docs/replication/replicators/index.rst 2012-03-18 00:36:19 +0000
@@ -0,0 +1,28 @@
1.. _replicators:
2
3.. _replication_replicators:
4
5Replicators
6===========
7
8Replicators are one end point of a
9:ref:`replication stream <replication_streams>` which provide an interface
10between the Drizzle kernel and :ref:`appliers`. The Drizzle kernel sends
11replication events, encoded as Google Buffer Protocol message, to every
12replicator that was loaded when Drizzled was stared.
13Replicators implement specific behaviors by modifying or filtering
14replication events before sending them to their paired appliers.
15For example, a replicator may filter replication events from specific schemas
16or tables. There are no restrictions on how a replicator can modify
17replication events.
18
19Replicators are implemented by plugins and have unique names. Most appliers
20can be configured to use a specific replicator by specifying its unique name.
21
22Drizzle includes the following replicator plugins:
23
24.. toctree::
25 :maxdepth: 1
26 :glob:
27
28 *
029
=== modified file 'plugin/default_replicator/docs/index.rst'
--- plugin/default_replicator/docs/index.rst 2011-10-23 05:45:09 +0000
+++ plugin/default_replicator/docs/index.rst 2012-03-18 00:36:19 +0000
@@ -1,66 +1,4 @@
1.. _default_replicator_plugin:
2
3Default Replicator1Default Replicator
4==================2==================
53
6:program:`default_replicator` is a simple replicator which replicates all4See :ref:`default_replicator`.
7write events to all appliers.
8
9.. seealso:: :doc:`/replication/drizzle`
10
11.. _default_replicator_loading:
12
13Loading
14-------
15
16This plugin is loaded by default, but it may need to be configured. See
17the plugin's :ref:`default_replicator_configuration` and
18:ref:`default_replicator_variables`.
19
20To stop the plugin from loading by default, start :program:`drizzled`
21with::
22
23 --plugin-remove=default_replicator
24
25.. seealso:: :ref:`drizzled_plugin_options` for more information about adding and removing plugins.
26
27.. _default_replicator_configuration:
28
29Configuration
30-------------
31
32This plugin does not have any command line options.
33
34.. _default_replicator_variables:
35
36Variables
37---------
38
39This plugin does not register any variables.
40
41.. _default_replicator_authors:
42
43Authors
44-------
45
46Jay Pipes
47
48.. _default_replicator_version:
49
50Version
51-------
52
53This documentation applies to **default_replicator 1.0**.
54
55To see which version of the plugin a Drizzle server is running, execute:
56
57.. code-block:: mysql
58
59 SELECT MODULE_VERSION FROM DATA_DICTIONARY.MODULES WHERE MODULE_NAME='default_replicator'
60
61Changelog
62---------
63
64v1.0
65^^^^
66* First release.
675
=== modified file 'plugin/filtered_replicator/docs/index.rst'
--- plugin/filtered_replicator/docs/index.rst 2011-10-23 05:45:09 +0000
+++ plugin/filtered_replicator/docs/index.rst 2012-03-18 00:36:19 +0000
@@ -1,145 +1,4 @@
1.. _filtered_replicator_plugin:
2
3Filtered Replicator1Filtered Replicator
4===================2===================
53
6The Filtered Replicator plugin registers itself in the Drizzle kernel4See :ref:`filtered_replicator`.
7replication stream process as a new replicator (see :ref:`replication_streams`
8for more information). It provides a way to filter replication messages by
9schema name or table name. Regular expressions can be used for the schema and
10table names.
11
12.. _filtered_replicator_loading:
13
14Loading
15-------
16
17To load this plugin, start :program:`drizzled` with::
18
19 --plugin-add=filtered_replicator
20
21Loading the plugin may not enable or configure it. See the plugin's
22:ref:`filtered_replicator_configuration` and :ref:`filtered_replicator_variables`.
23
24.. seealso:: :ref:`drizzled_plugin_options` for more information about adding and removing plugins.
25
26.. _filtered_replicator_configuration:
27
28Configuration
29-------------
30
31These command line options configure the plugin when :program:`drizzled`
32is started. See :ref:`command_line_options` for more information about specifying
33command line options.
34
35.. program:: drizzled
36
37.. option:: --filtered-replicator.filteredschemas ARG
38
39 :Default:
40 :Variable: :ref:`filtered_replicator_filteredschemas <filtered_replicator_filteredschemas>`
41
42 Comma-separated list of schemas to exclude from replication.
43
44.. option:: --filtered-replicator.filteredtables ARG
45
46 :Default:
47 :Variable: :ref:`filtered_replicator_filteredtables <filtered_replicator_filteredtables>`
48
49 Comma-separated list of tables to exclude from replication.
50
51.. option:: --filtered-replicator.schemaregex ARG
52
53 :Default:
54 :Variable: :ref:`filtered_replicator_schemaregex <filtered_replicator_schemaregex>`
55
56 Regular expression to apply to schemas to exclude from replication.
57
58.. option:: --filtered-replicator.tableregex ARG
59
60 :Default:
61 :Variable: :ref:`filtered_replicator_tableregex <filtered_replicator_tableregex>`
62
63 Regular expression to apply to tables to exclude from replication.
64
65.. _filtered_replicator_variables:
66
67Variables
68---------
69
70These variables show the running configuration of the plugin.
71See `variables` for more information about querying and setting variables.
72
73.. _filtered_replicator_filteredschemas:
74
75* ``filtered_replicator_filteredschemas``
76
77 :Scope: Global
78 :Dynamic: No
79 :Option: :option:`--filtered-replicator.filteredschemas`
80
81 Comma-separated list of schemas to exclude from replication.
82
83.. _filtered_replicator_filteredtables:
84
85* ``filtered_replicator_filteredtables``
86
87 :Scope: Global
88 :Dynamic: No
89 :Option: :option:`--filtered-replicator.filteredtables`
90
91 Comma-separated list of tables to exclude from replication.
92
93.. _filtered_replicator_schemaregex:
94
95* ``filtered_replicator_schemaregex``
96
97 :Scope: Global
98 :Dynamic: No
99 :Option: :option:`--filtered-replicator.schemaregex`
100
101 Regular expression to apply to schemas to exclude from replication.
102
103.. _filtered_replicator_tableregex:
104
105* ``filtered_replicator_tableregex``
106
107 :Scope: Global
108 :Dynamic: No
109 :Option: :option:`--filtered-replicator.tableregex`
110
111 Regular expression to apply to tables to exclude from replication.
112
113.. _filtered_replicator_examples:
114
115Examples
116--------
117
118Sorry, there are no examples for this plugin.
119
120.. _filtered_replicator_authors:
121
122Authors
123-------
124
125Padraig O Sullivan
126
127.. _filtered_replicator_version:
128
129Version
130-------
131
132This documentation applies to **filtered_replicator 0.2**.
133
134To see which version of the plugin a Drizzle server is running, execute:
135
136.. code-block:: mysql
137
138 SELECT MODULE_VERSION FROM DATA_DICTIONARY.MODULES WHERE MODULE_NAME='filtered_replicator'
139
140Chagnelog
141---------
142
143v0.2
144^^^^
145* First release.
1465
=== modified file 'plugin/innobase/docs/index.rst'
--- plugin/innobase/docs/index.rst 2012-03-08 00:54:48 +0000
+++ plugin/innobase/docs/index.rst 2012-03-18 00:36:19 +0000
@@ -49,10 +49,10 @@
4949
50.. _innodb_transaction_log:50.. _innodb_transaction_log:
5151
52InnoDB Transaction Log52InnoDB Replicaiton Log
53----------------------53----------------------
5454
55The :program:`innodb` plugin provides a mechanism to store replication55The ``innodb`` plugin provides a mechanism to store replication
56events in an InnoDB table. When enabled, this transaction log can be accessed56events in an InnoDB table. When enabled, this transaction log can be accessed
57through the SYS_REPLICATION_LOG and INNODB_REPLICATION_LOG tables in the57through the SYS_REPLICATION_LOG and INNODB_REPLICATION_LOG tables in the
58DATA_DICTIONARY schema.58DATA_DICTIONARY schema.
@@ -524,6 +524,13 @@
524524
525 Use InnoDB's internal memory allocator instead of the system's malloc.525 Use InnoDB's internal memory allocator instead of the system's malloc.
526526
527.. option:: --innodb.use-replicator
528
529 :Default: default
530 :Variable: `innodb_use_replicator <innodb_use_sys_malloc>`
531
532 Use this replicator for the :ref:`innodb_transaction_log`.
533
527.. option:: --innodb.version ARG534.. option:: --innodb.version ARG
528535
529 :Default:536 :Default:
@@ -1110,6 +1117,16 @@
11101117
1111 If system or internal malloc() is being used.1118 If system or internal malloc() is being used.
11121119
1120.. _innodb_use_replicator:
1121
1122* ``innodb_use_replicator``
1123
1124 :Scope: Global
1125 :Dynamic: No
1126 :Option: :option:`--innodb.use-replicator`
1127
1128 Replicator to which the :ref:`innodb_transaction_log` is paired.
1129
1113.. _innodb_version_var:1130.. _innodb_version_var:
11141131
1115* ``innodb_version``1132* ``innodb_version``
11161133
=== modified file 'plugin/rabbitmq/docs/index.rst'
--- plugin/rabbitmq/docs/index.rst 2011-10-23 05:45:09 +0000
+++ plugin/rabbitmq/docs/index.rst 2012-03-18 00:36:19 +0000
@@ -1,243 +1,6 @@
1.. _rabbitmq_plugin:1.. _rabbitmq_plugin:
22
3RabbitMQ Integration3RabbitMQ Integration
4======================4====================
55
6It is possible to replicate transactions directly to a `RabbitMQ <http://www.rabbitmq.org>`_ server from drizzle, this could be used to create advanced replication solutions, to visualize data, or to build triggers. For example, `RabbitReplication <http://www.rabbitreplication.org>`_ has been built to consume the messages from rabbitmq, transform them, and persist the data in drizzle, mysql, many nosql stores and even replicating directly to websockets for data visualization.6See :ref:`rabbitmq_applier`.
7
8Loading
9-------
10
11To load this plugin, start :program:`drizzled` with::
12
13 --plugin-add=rabbitmq
14
15Loading the plugin may not enable or configure it. See the plugin's
16:ref:`rabbitmq_configuration` and :ref:`rabbitmq_variables`.
17
18.. seealso:: :ref:`drizzled_plugin_options` for more information about adding and removing plugins.
19
20.. _rabbitmq_configuration:
21
22Configuration
23-------------
24
25These command line options configure the plugin when :program:`drizzled`
26is started. See :ref:`command_line_options` for more information about specifying
27command line options.
28
29.. program:: drizzled
30
31.. option:: --rabbitmq.exchange ARG
32
33 :Default: ReplicationExchange
34 :Variable: :ref:`rabbitmq_exchange <rabbitmq_exchange>`
35
36 Name of RabbitMQ exchange to publish to
37
38.. option:: --rabbitmq.host ARG
39
40 :Default: localhost
41 :Variable: :ref:`rabbitmq_host <rabbitmq_host>`
42
43 Host name to connect to
44
45.. option:: --rabbitmq.password ARG
46
47 :Default: guest
48 :Variable: :ref:`rabbitmq_password <rabbitmq_password>`
49
50 RabbitMQ password
51
52.. option:: --rabbitmq.port ARG
53
54 :Default: 5672
55 :Variable: :ref:`rabbitmq_port <rabbitmq_port>`
56
57 Port to connect to
58
59.. option:: --rabbitmq.routingkey ARG
60
61 :Default: ReplicationRoutingKey
62 :Variable: :ref:`rabbitmq_routingkey <rabbitmq_routingkey>`
63
64 Name of RabbitMQ routing key to use
65
66.. option:: --rabbitmq.use-replicator ARG
67
68 :Default: default_replicator
69 :Variable:
70
71 Name of the replicator plugin to use (default='default_replicator')
72
73.. option:: --rabbitmq.username ARG
74
75 :Default: guest
76 :Variable: :ref:`rabbitmq_username <rabbitmq_username>`
77
78 RabbitMQ username
79
80.. option:: --rabbitmq.virtualhost ARG
81
82 :Default: /
83 :Variable: :ref:`rabbitmq_virtualhost <rabbitmq_virtualhost>`
84
85 RabbitMQ virtualhost
86
87.. _rabbitmq_variables:
88
89Variables
90---------
91
92These variables show the running configuration of the plugin.
93See `variables` for more information about querying and setting variables.
94
95.. _rabbitmq_exchange:
96
97* ``rabbitmq_exchange``
98
99 :Scope: Global
100 :Dynamic: No
101 :Option: :option:`--rabbitmq.exchange`
102
103 Name of RabbitMQ exchange to publish to
104
105.. _rabbitmq_host:
106
107* ``rabbitmq_host``
108
109 :Scope: Global
110 :Dynamic: No
111 :Option: :option:`--rabbitmq.host`
112
113 Host name to connect to
114
115.. _rabbitmq_password:
116
117* ``rabbitmq_password``
118
119 :Scope: Global
120 :Dynamic: No
121 :Option: :option:`--rabbitmq.password`
122
123 RabbitMQ password
124
125.. _rabbitmq_port:
126
127* ``rabbitmq_port``
128
129 :Scope: Global
130 :Dynamic: No
131 :Option: :option:`--rabbitmq.port`
132
133 Port to connect to
134
135.. _rabbitmq_routingkey:
136
137* ``rabbitmq_routingkey``
138
139 :Scope: Global
140 :Dynamic: No
141 :Option: :option:`--rabbitmq.routingkey`
142
143 Name of RabbitMQ routing key to use
144
145.. _rabbitmq_username:
146
147* ``rabbitmq_username``
148
149 :Scope: Global
150 :Dynamic: No
151 :Option: :option:`--rabbitmq.username`
152
153 RabbitMQ username
154
155.. _rabbitmq_virtualhost:
156
157* ``rabbitmq_virtualhost``
158
159 :Scope: Global
160 :Dynamic: No
161 :Option: :option:`--rabbitmq.virtualhost`
162
163 RabbitMQ virtualhost
164
165.. _rabbitmq_examples:
166
167Examples
168--------
169
170First install a recent version of RabbitMQ, then install librabbitmq, the C library for talking to the RabbitMQ server:
171
172.. code-block:: bash
173
174 $ hg clone http://hg.rabbitmq.com/rabbitmq-codegen/
175 $ hg clone http://hg.rabbitmq.com/rabbitmq-c/
176 $ cd rabbitmq-c
177 $ autoreconf -f -i
178 $ ./configure
179 $ make
180 $ make install
181
182Now you probably need to rebuild Drizzle since the :program:`rabbitmq` plugin is not built if librabbitmq is not installed.
183
184Finally, start :program:`drizzled` like:
185
186.. code-block:: bash
187
188 sbin/drizzled --plugin-add rabbitmq,default-replicator \
189 --rabbitmq.use-replicator default
190
191To verify that it works, you can start a generic rabbitmq listener from librabbitmq:
192
193.. code-block:: bash
194
195 $ amqp_listen localhost 5672 ReplicationExchange ReplicationRoutingKey
196
197And you should see something like this when you do an INSERT/CREATE/.. (just not a select) in your newly built Drizzle instance::
198
199 Result 0
200 Frame type 1, channel 1
201 Method AMQP_BASIC_DELIVER_METHOD
202 Delivery 1, exchange ReplicationExchange routingkey ReplicationRoutingKey
203
204 00000000: 0A 17 08 01 10 87 36 18 : F0 FA D9 99 FA F1 A7 02 ......6.........
205 00000010: 20 99 81 DA 99 FA F1 A7 : 02 12 40 08 01 10 F2 FA .........@.....
206 00000020: D9 99 FA F1 A7 02 18 FC : FA D9 99 FA F1 A7 02 2A ...............*
207 00000030: 17 0A 06 0A 01 62 12 01 : 61 12 06 08 04 12 02 69 .....b..a......i
208 00000040: 64 12 05 08 01 12 01 74 : 32 11 08 01 10 01 1A 0B d......t2.......
209 00000050: 0A 01 32 0A 02 61 61 10 : 00 10 00 20 01 28 01 ..2..aa.... .(.
210 0000005F:
211
212Implementation Details
213----------------------
214
215* :program:`drizzled` will not sart if the rabbitmq server is not available.
216* If the rabbitmq server goes away, the plugin will try to reconnect and resend the message 3 times, after that, the transaction is rolled back.
217
218.. _rabbitmq_authors:
219
220Authors
221-------
222
223Marcus Eriksson
224
225.. _rabbitmq_version:
226
227Version
228-------
229
230This documentation applies to **rabbitmq 0.1**.
231
232To see which version of the plugin a Drizzle server is running, execute:
233
234.. code-block:: mysql
235
236 SELECT MODULE_VERSION FROM DATA_DICTIONARY.MODULES WHERE MODULE_NAME='rabbitmq'
237
238Changelog
239---------
240
241v0.1
242^^^^
243* First release.
2447
=== removed file 'plugin/slave/docs/admin.rst'
--- plugin/slave/docs/admin.rst 2011-03-11 18:38:09 +0000
+++ plugin/slave/docs/admin.rst 1970-01-01 00:00:00 +0000
@@ -1,136 +0,0 @@
1********************************
2Replication Slave Administration
3********************************
4
5This page walks you through some common administration tasks when using
6the replication slave plugin.
7
8Monitoring the Master
9#####################
10
11Slave Connections
12*****************
13
14If you want to determine which slave machines are connected to your
15master, use the *SHOW PROCESSLIST* command. Slave connections will show
16up in the output of this command.
17
18InnoDB Transaction Log
19**********************
20
21The slave plugin uses the InnoDB transaction log (see
22:ref:`innodb_transaction_log`) on the master to retrieve replication
23messages. This transaction log, though stored as an internal table within
24InnoDB, offers two different views to the table contents. Two tables in
25the DATA_DICTIONARY schema provide the different views into the transaction
26log: the SYS_REPLICATION_LOG table and the INNODB_REPLICATION_LOG table.
27
28The SYS_REPLICATION_LOG table is read directly by the slave plugin.
29This table is described as below::
30
31 drizzle> SHOW CREATE TABLE data_dictionary.sys_replication_log\G
32 *************************** 1. row ***************************
33 Table: SYS_REPLICATION_LOG
34 Create Table: CREATE TABLE `SYS_REPLICATION_LOG` (
35 `ID` BIGINT,
36 `SEGID` INT,
37 `COMMIT_ID` BIGINT,
38 `END_TIMESTAMP` BIGINT,
39 `MESSAGE_LEN` INT,
40 `MESSAGE` BLOB,
41 PRIMARY KEY (`ID`,`SEGID`) USING BTREE,
42 KEY `COMMIT_IDX` (`COMMIT_ID`,`ID`) USING BTREE
43 ) ENGINE=InnoDB COLLATE = binary
44
45The INNODB_REPLICATION_LOG is similar to the SYS_REPLICATION_LOG, the
46main difference being that the Google Protobuffer message representing
47the changed rows is converted to plain text before being output::
48
49 drizzle> SHOW CREATE TABLE data_dictionary.innodb_replication_log\G
50 *************************** 1. row ***************************
51 Table: INNODB_REPLICATION_LOG
52 Create Table: CREATE TABLE `INNODB_REPLICATION_LOG` (
53 `TRANSACTION_ID` BIGINT NOT NULL,
54 `TRANSACTION_SEGMENT_ID` BIGINT NOT NULL,
55 `COMMIT_ID` BIGINT NOT NULL,
56 `END_TIMESTAMP` BIGINT NOT NULL,
57 `TRANSACTION_MESSAGE_STRING` TEXT COLLATE utf8_general_ci NOT NULL,
58 `TRANSACTION_LENGTH` BIGINT NOT NULL
59 ) ENGINE=FunctionEngine COLLATE = utf8_general_ci REPLICATE = FALSE
60
61The INNODB_REPLICATION_LOG table is read-only due to the way it is
62implemented. The SYS_REPLICATION_LOG table, on the other hand, allows you
63to modify the contents of the transaction log. You would use this table
64to trim the transaction log.
65
66Monitoring the Slave
67####################
68
69The slave plugin has two types of threads doing all of the work:
70
71* An IO (or producer) thread
72* An applier (or consumer) thread
73
74The status of each thread is stored in tables in the *sys_replication*
75schema. The IO thread status is contained in the *io_state* table, and
76the applier thread status is in the *applier_state* table. You may query
77these tables just like any other table. For example::
78
79 drizzle> SELECT * FROM sys_replication.io_state\G
80 *************************** 1. row ***************************
81 status: RUNNING
82 error_msg:
83
84The above shows that the IO thread is **RUNNING**. If there had been
85an error on the IO thread, the status value would be **STOPPED** and
86the error_msg column would contain information about the error.
87
88We can check the state of the applier thread in a similar manner::
89
90 drizzle> SELECT * FROM sys_replication.applier_state\G
91 *************************** 1. row ***************************
92 last_applied_commit_id: 4
93 status: RUNNING
94 error_msg:
95
96The status and error_msg columns are similar to the ones in the *io_state*
97table. Also available is the last_applied_commit_id, which contains the
98value of the COMMIT_ID from the master's replication log (see definition
99of the data_dictionary.sys_replication_log table above) of the most
100recently executed transaction.
101
102Transaction Log Maintenance
103###########################
104
105Currently, the InnoDB transaction log grows without bounds and is never
106trimmed of unneeded entries. This can present a problem for long running
107replication setups. You may trim the log manually, but you must make certain
108to not remove any entries that are needed by slave servers.
109
110Follow these steps to trim the InnoDB transaction without affecting slave
111function:
112
113#. Query each slave for the *last_applied_commit_id* value from the *sys_replication.applier_state* table.
114#. Choose the **minimum** value obtained from step one. This will be the marker for the slave that is the furthest behind the master.
115#. Using this marker value from the previous step, delete all entries from the master's transaction log that has a COMMIT_ID less than the marker value.
116
117Below is an example of the steps defined above. First, step 1 and 2. Assume
118that we have two slave hosts connected to the master (slave-1 and slave-2).
119We need to query both to check their relationship with the master transaction
120log::
121
122 slave-1> SELECT last_applied_commit_id FROM sys_replication.applier_state\G
123 *************************** 1. row ***************************
124 last_applied_commit_id: 3000
125
126 slave-2> SELECT last_applied_commit_id FROM sys_replication.applier_state\G
127 *************************** 1. row ***************************
128 last_applied_commit_id: 2877
129
130We see that slave-2 has the smallest value for *last_applied_commit_id*. We
131will use this value in the next step to trim the transaction log on the
132master::
133
134 master> DELETE FROM data_dictionary.sys_replication_log WHERE commit_id < 2877;
135
136This will remove all old, unneeded entries from the InnoDB transaction log. Note that the SYS_REPLICATION_LOG table is used for this maintenance task.
1370
=== modified file 'plugin/slave/docs/index.rst'
--- plugin/slave/docs/index.rst 2012-03-08 00:54:48 +0000
+++ plugin/slave/docs/index.rst 2012-03-18 00:36:19 +0000
@@ -1,165 +1,4 @@
1.. _slave_plugin:
2
3Replication Slave1Replication Slave
4==================2==================
53
6This page contains the configuration and implementation details for Drizzle replication.
7
8See these pages for user-level information:
9
10.. toctree::
11 :maxdepth: 2
12
13 user_example
14 admin
15
16Description
17-----------
18
19Replication enables data from one Drizzle database server (the master) to be replicated to one or more Drizzle database servers (the slaves). It provides a native implementation of replication between two Drizzle processes. Depending on your configuration, you can replicate all databases, selected databases, or selected tables within a database.
20
21Drizzle’s replication system is entirely new and different from MySQL. Replication events are stored using Google Protocol Buffer messages in an InnoDB table. These events are read by the slave, stored locally, and applied. The advantage of the Google Protocol Buffer messages is a script or program can be put together in pretty much any language in minutes, and read the replication log. In other words, replication plugins are easy to implement, which enable developers to entirely customize their replication system.
22
23This plugin requires a master that is running with the InnoDB replication log enabled. The slave plugin allows a server to replicate from another server that is using the innodb-trx log. It is a very simple setup:
244
25master options:  –innodb.replication-log=true
26slave options: –plugin-add=slave –slave.config-file=XYZ5slave options: –plugin-add=slave –slave.config-file=XYZ
27
28The config file may contain the following options in option=value format:
296
30master-host – hostname/ip of the master host
31master-port – port used by the master server7master-port – port used by the master server
32master-user – username8master-user – username
33master-pass – password9master-pass – password
34max-reconnects – max # of reconnect attempts if the master disappears10max-reconnects – max # of reconnect attempts if the master disappears
35seconds-between-reconnects – how long to wait between reconnect attempts11seconds-between-reconnects – how long to wait between reconnect attempts
3612See :ref:`slave_applier`.
37
38Configuration
39-------------
40
41Most of the options that can be used to control the replication slave plugin
42can only be given in a configuration file. The only exceptions are the
43:option:`drizzled --slave.config-file` and :option:`drizzled --slave.max-commit-id` options.
44
45.. program:: drizzled
46
47.. option:: --slave.config-file FILE
48
49 :Default: :file:`etc/slave.cfg`
50 :Variable:
51
52 Path to the replication slave configuration file. By default, the
53 plugin will look for a file named :file:`slave.cfg` in the `etc` directory
54 of the Drizzle installation. If you want to specify a different path or
55 configuration file name, it is best to specify a full path to the
56 file. The relative path used by plugins is within the :option:`--datadir`
57 directory, so a full path is recommended.
58
59.. option:: --slave.max-commit-id ID
60
61 Manually set the maximum commit ID the slave is assumed to have retrieved
62 from the master. This value will be used by the slave to determine where
63 to begin retrieving replication messages from the master transaction log.
64 This option can be used to provision a new slave machine by setting it to
65 the value output from the drizzledump client when used with the
66 --single-transaction option.
67
68 This value is not allowed to be set via the configuration file since
69 you would normally only set it once on initial slave startup. This
70 eliminates the possibility of forgetting to delete it from the configuration
71 file for subsequent slave restarts.
72
73The options below are read from the configuration file.
74
75.. confval:: master-host
76
77 Hostname/IP address of the master server.
78
79.. confval:: master-port
80
81 Drizzle port used by the master server. Default is 3306.
82
83.. confval:: master-user
84
85 Username to use for connecting to the master server.
86
87.. confval:: master-pass
88
89 Password associated with the username given by :confval:`master-user`.
90
91.. confval:: max-reconnects
92
93 The number of reconnection attempts the slave plugin will try if the
94 master server becomes unreachable. Default is 10.
95
96.. confval:: seconds-between-reconnects
97
98 The number of seconds to wait between reconnect attempts when the master
99 server becomes unreachable. Default is 30.
100
101.. confval:: io-thread-sleep
102
103 The number of seconds the IO (producer) thread sleeps between queries to the
104 master for more replication events. Default is 5.
105
106.. confval:: applier-thread-sleep
107
108 The number of seconds the applier (consumer) thread sleeps between applying
109 replication events from the local queue. Default is 5.
110
111Implementation Details
112----------------------
113
114The replication slave plugin creates two worker threads, each accessing a
115work queue (implemented as an InnoDB table) that contains the replication
116events. This is a producer/consumer paradigm where one thread populates the
117queue (the producer), and the other thread (the consumer) reads events from
118the queue.
119
120The producer thread (or I/O thread) is in charge of connecting to the master
121server and pulling down replication events from the master's transaction
122log and storing them locally in the slave queue. It is required that the
123master use the InnoDB replication log (:option:`--innodb.replication-log <drizzled --innodb.replication-log>`).
124
125The consumer thread (or applier thread) reads the replication events from
126the local slave queue, applies them locally, and then deletes successfully
127applied events from the queue.
128
129Schemas and Tables
130------------------
131
132The slave plugin creates its own schema and set of tables to store its
133metadata. It stores everything in the **sys_replication** schema. The
134following are the tables that it will create:
135
136.. dbtable:: sys_replication.io_state
137
138 Stores metadata about the IO/producer thread.
139
140.. dbtable:: sys_replication.applier_state
141
142 Stores metadata about the applier/consumer thread.
143
144.. dbtable:: sys_replication.queue
145
146 The replication event queue.
147
148.. _slave_authors:
149
150Authors
151-------
152
153David Shrewsbury
154
155.. _slave_version:
156
157Version
158-------
159
160This documentation applies to **slave 1.0**.
161
162To see which version of the plugin a Drizzle server is running, execute:
163
164.. code-block:: mysql
165
166 SELECT MODULE_VERSION FROM DATA_DICTIONARY.MODULES WHERE MODULE_NAME='slave'
167
168Changelog
169---------
170
171v1.0
172^^^^
173* First release.
17413
=== removed file 'plugin/slave/docs/user_example.rst'
--- plugin/slave/docs/user_example.rst 2011-07-22 23:56:42 +0000
+++ plugin/slave/docs/user_example.rst 1970-01-01 00:00:00 +0000
@@ -1,134 +0,0 @@
1How to use replication: An example
2====================================
3
4A simple replication setup (using a single master and a single slave) between two Drizzle servers is done with the replication slave plugin. With Drizzle replication, you can also provision a new slave into an existing setup.
5
6Replication setup begins with making certain that both master and slave share the same version of Drizzle to avoid any potential incompatibility issues.
7
8Master Setup
9-------------
10
11Setting up the master is the first step. An important requirement is to start the master Drizzle database server with the --innodb.replication-log option, and a few other options in most circumstances. More options can be found in the options documentation. These are the most common options needed for a replication master. For example:
12
13 master> usr/local/sbin/drizzled \
14 --innodb.replication-log \
15 --pid-file=/var/run/drizzled/drizzled.pid \
16 --drizzle-protocol.bind-address=0.0.0.0 \
17 --mysql-protocol.bind-address=0.0.0.0 \
18 --daemon
19
20
21Several options are required on most setups. They are set on Drizzle Startup with a --optionname. The most important ones are:
22
23
24The InnoDB replication log must be running:
25
26--innodb.replication-log
27
28
29PID must be set:
30
31--pid-file=/var/run/drizzled/drizzled.pid
32
33
34the address binding for Drizzle's default port (4427):
35
36--drizzle-protocol.bind-address=0.0.0.0
37
38
39The address binding for systems replicating through MySQL's default port (3306):
40
41--mysql-protocol.bind-address=0.0.0.0
42
43
44Data Directory can be set other than default:
45
46--datadir=$PWD/var
47
48
49For more complex setups, the server id option may be appropriate to use:
50
51--server-id
52
53
54To run Drizzle in the background, thereby keeping the database running if the user logs out:
55
56--daemon
57
58
59
60With the master running, you can optionally now create a backup of any databases to be imported on the new slave by using :doc:`../../clients/drizzledump`. This example, however, assumes that we are starting with a fresh database with no data.
61
62Slave Setup
63-------------
64
65Starting the slave is very similar to starting the master. There are two Drizzle database server options required for the slave: --plugin-add=slave and --slave.config-file. For example: ::
66
67 slave> /usr/local/sbin/drizzled \
68 --plugin-add=slave \
69 --slave.config-file=/usr/local/etc//slave.cfg
70
71A more typical startup will need more options:
72
73 slave> /usr/local/sbin/drizzled \
74 --plugin-add=slave \
75 --slave.config-file=/usr/local/etc//slave.cfg \
76 --pid-file=/var/run/drizzled/drizzled.pid \
77 --drizzle-protocol.bind-address=0.0.0.0 \
78 --mysql-protocol.bind-address=0.0.0.0 \
79 --daemon
80
81Similar to the Master setup, there are a number of options that can be selected. Please see the Plugin Documentation for the relevent options.
82
83These options tell the server to load the slave plugin, and then tell the slave plugin where to find the slave host configuration file. This configuration file has options to specify the master host and a few options to control how the slave operates. You can read more about the available configuration options in the replication slave plugin documentation. Below is a simple example: ::
84
85 master-host = master.location.com
86 master-port = 4427
87
88Some options that can be set other than default, but are otherwise not necessary, are:
89
90 master-user = dino_slave
91 master-pass = my_password
92 io-thread-sleep = 10
93 applier-thread-sleep = 10
94
95The slave will immediately connect to the master host specified in the configuration file and begin pulling events from the InnoDB-based transaction log. By default, a freshly provisioned slave will begin pulling from the beginning of this transaction log. Once all replication messages have been pulled from the master and stored locally on the slave host, the IO thread will sleep and periodically awaken to check for more messages. This is straightforward for an initial replication setup. See below to learn about inserting another slave host into an already existing replication architecture.
96
97Provisioning a New Slave Host
98-------------------------------
99
100The basic formula for creating a new slave host for an existing replication setup is:
101
102 1. Make a backup of the master databases.
103 2. Record the state of the master transaction log at the point the backup was made.
104 3. Restore the backup on the new slave machine.
105 4. Start the new slave and tell it to begin reading the transaction log from the point recorded in #2.
106
107Steps #1 and #2 are covered with the drizzledump client program. If you use the --single-transaction option to drizzledump, it will place a comment near the beginning of the dump output with the InnoDB transaction log metadata. For example: ::
108
109 master> drizzledump --all-databases --single-transaction > master.backup
110 master> head -1 master.backup
111 -- SYS_REPLICATION_LOG: COMMIT_ID = 33426, ID = 35074
112
113The SYS_REPLICATION_LOG line provides the replication log metadata needed when starting a new slave. It has two pieces of information:
114
115* **COMMIT_ID**: This value is the commit sequence number recorded for the most recently executed transaction stored in the transaction log. We can use this value to determine proper commit order within the log. The unique transaction ID cannot be used since that value is assigned when the transaction is started, not when it is committed.
116* **ID**: This is the unique transaction identifier associated with the most recently executed transaction stored in the transaction log.
117
118Next, steps #3 and #4 must be completed to start the new slave. First, you must start the slave WITHOUT the replication slave plugin enabled, to prevent it from reading from the master until the backup is imported. To start it without the plugin enabled, import your backup, then shutdown the server: ::
119
120 slave> sbin/drizzled --datadir=$PWD/var &
121 slave> drizzle < master.backup
122 slave> drizzle --shutdown
123
124Now that the backup is imported, restart the slave with the replication slave plugin enabled and use a new option, --slave.max-commit-id, to force the slave to begin reading the master's transaction log at the proper location:
125
126 slave> sbin/drizzled --datadir=$PWD/var \
127 --plugin-add=slave \
128 --slave.config-file=/user/local/etc/slave.cfg \
129 --slave.max-commit-id=33426 &
130
131
132We give the --slave.max-commit-id the value from the comment in the master dump file, which defines the maximum COMMIT_ID value (the latest transaction) represented by the slave's contents.
133
134This is the full cycle for a simple replication example. Please see the other Drizzle slave plugin docs for more information on replication and configuration options.
1350
=== modified file 'plugin/slave/module.cc'
--- plugin/slave/module.cc 2012-01-15 20:54:59 +0000
+++ plugin/slave/module.cc 2012-03-18 00:36:19 +0000
@@ -62,7 +62,7 @@
62{62{
63 DRIZZLE_VERSION_ID,63 DRIZZLE_VERSION_ID,
64 "slave",64 "slave",
65 "1.0",65 "1.1",
66 "David Shrewsbury",66 "David Shrewsbury",
67 N_("Drizzle replication slave"),67 N_("Drizzle replication slave"),
68 PLUGIN_LICENSE_GPL,68 PLUGIN_LICENSE_GPL,
6969
=== modified file 'plugin/zeromq/docs/index.rst'
--- plugin/zeromq/docs/index.rst 2011-10-23 05:45:09 +0000
+++ plugin/zeromq/docs/index.rst 2012-03-18 00:36:19 +0000
@@ -3,51 +3,4 @@
3ZeroMQ3ZeroMQ
4======4======
55
6ZeroMQ is a messaging library that allows you to easily build complex6See :ref:`zeromq_applier`.
7communication systems. The ZeroMQ plugin allows drizzle to publish
8transactions to a local PUB socket. Many clients can subscribe to
9these transactions. The first frame of the message sent out is the
10schema name the transaction touched - this enables clients to only
11subscribe to the interesting schemas (note that certain "transactions"
12are without a schema, like SET xyz for example, for these, the first
13frame is empty).
14
15Getting started
16---------------
17
18First, install zeromq, get the code from `zeromq.org
19<http://zeromq.org>`_, then you can build drizzle, watch the
20./configure output to verify that drizzle finds the libraries needed.
21
22Now you are good to go, simply start drizzle with --plugin-add=zeromq
23and drizzle will start publishing transactions. The only configuration
24parameter available is:
25
26.. code-block:: bash
27
28 --zeromq.endpoint arg (=tcp://*:9999) - the endpoint to expose.
29
30Now you can write a simple python script to verify that it works,
31something like this will work:
32
33.. code-block:: python
34
35 import zmq
36
37 ctx = zmq.Context()
38 s = ctx.socket(zmq.SUB)
39 s.setsockopt(zmq.SUBSCRIBE, '')
40 s.connect('tcp://localhost:9999')
41 i = 0
42 while True:
43 i = i+1
44 s.recv_multipart()
45 print i
46
47and then you can generate some load:
48
49.. code-block:: bash
50
51 bin/drizzleslap -c 10 -a --auto-generate-sql-add-autoincrement --burnin
52
53which creates 10 threads and generates random queries.