Drizzle

Merge lp:~sharan-monikantan/drizzle/qp-sysbench-gsoc into lp:drizzle

  1. qp-sysbench-gsoc
  2. Merge into 7.2
Proposed by Sharan Kumar
Status: Merged
Approved by: Patrick Crews
Approved revision: 2583
Merged at revision: 2584
Proposed branch: lp:~sharan-monikantan/drizzle/qp-sysbench-gsoc
Merge into: lp:drizzle
Diff against target: 1922 lines (+1425/-141)
20 files modified
docs/testing/kewpie.rst (+46/-6)
docs/testing/sysbench.rst (+130/-77)
tests/lib/modes/native/native_test_execution.py (+2/-0)
tests/lib/opts/test_run_options.py (+18/-0)
tests/lib/server_mgmt/drizzled.py (+3/-2)
tests/lib/server_mgmt/server.py (+26/-0)
tests/lib/sys_mgmt/system_management.py (+1/-0)
tests/lib/util/crashme_methods.py (+7/-0)
tests/lib/util/database_connect.py (+62/-0)
tests/lib/util/drizzleslap_methods.py (+113/-0)
tests/lib/util/mailing_report.py (+44/-0)
tests/lib/util/mysql_methods.py (+16/-6)
tests/lib/util/sysbenchTestCase.py (+156/-0)
tests/lib/util/sysbench_methods.py (+349/-2)
tests/qp_tests/crashme/crashme_test.py (+27/-5)
tests/qp_tests/drizzleslap/drizzleslap_test.py (+195/-0)
tests/qp_tests/sqlbench/sqlbench_test.py (+6/-0)
tests/qp_tests/sysbench/sysbench_readonly_test.py (+28/-43)
tests/qp_tests/sysbench/sysbench_readwrite_test.py (+89/-0)
tests/std_data/sysbench_db.sql (+107/-0)
To merge this branch: bzr merge lp:~sharan-monikantan/drizzle/qp-sysbench-gsoc
Reviewer Review Type Date Requested Status
Patrick Crews Approve
Review via email: mp+120625@code.launchpad.net

Description of the change

This merge is in connection with the work done on "Improved Performance Regression Monitoring" project (GSoC)
The following changes are made:
1. Added drizzle-automation functionality to kewpie
2. Imported SysBench to kewpie
3. Added several modules for the above mentioned tasks
4. Improved documentation

To post a comment you must log in.
Revision history for this message
Patrick Crews (patrick-crews) wrote :

Looks good to me. make distcheck passed.
sysbench is behaving like drizzle-automation for db interaction
we have regression-checking built-in (test will fail if tps percent_diff_from_avg <= -5%

review: Approve
Revision history for this message
Brian Aker (brianaker) wrote :

How do we enable this in Jenkins?

Revision history for this message
Patrick Crews (patrick-crews) wrote :

On 08/23/2012 01:38 AM, Brian Aker wrote:
> How do we enable this in Jenkins?
>

1) Make sure we have drizzle-sysbench + dbd::drizzle installed on the
node(s) we want to use

2) Make sure we have the results_db + tables we want created - sql
lives in /std_data/sysbench_db.sql

3) ./kewpie.py --suite=sysbench
--results-db-dsn="host:user:pw:schema:port"
[--mail-tgt=email_address_to_receive_report]

This kicks off the sysbench tests and will strore and analyze the
results via the results_db...basically the exact same queries, reports,
etc as drizzle-automation, but in-tree.

Might still need to tweak branch names, but other than that, easy-peasy.

Revision history for this message
Patrick Crews (patrick-crews) wrote :

On 08/23/2012 01:38 AM, Brian Aker wrote:
> How do we enable this in Jenkins?
>
 Ah, more accurately:
http://docs.drizzle.org/testing/sysbench.html

Thanks to shar for also producing pretty good docs : )

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'docs/testing/kewpie.rst'
2--- docs/testing/kewpie.rst 2012-03-08 00:54:48 +0000
3+++ docs/testing/kewpie.rst 2012-08-21 17:59:20 +0000
4@@ -59,6 +59,7 @@
5
6 * crashme - sql-bench's crashme suite (may take some time to run)
7 * sqlbench - sql-bench comprehensive suite. (may take ~45 min. to execute)
8+ * sysbench - SysBench database server performance (OLTP benchmark)
9
10
11 Running tests
12@@ -75,7 +76,7 @@
13 ------------------------
14 If one only wants to run a few, specific tests, they may do so this way::
15
16- ./kewpie.py [OPTIONS] test1 [test2 ... testN]
17+ ./kewpie.py [OPTIONS] test1 [test2 [test3 [...] ] ]
18
19 Running all tests within a suite
20 --------------------------------
21@@ -94,7 +95,7 @@
22 --------------------------------------
23 To run a specific set of tests within a suite::
24
25- ./kewpie.py [OPTIONS] --suite=SUITENAME TEST1 [TEST2..TESTN]
26+ ./kewpie.py [OPTIONS] --suite=SUITENAME test1 [test2 [test3 [...] ] ]
27
28 Calling tests using <suitename>.<testname> currently does not work. One must
29 specify the test suite via the :option:`kewpie.py --suite` option.
30@@ -107,7 +108,7 @@
31
32 Otherwise, one should simply name all suites::
33
34- ./kewpie.py [OPTIONS] --suite=SUITE1, SUITE2, ...SUITEN
35+ ./kewpie.py [OPTIONS] --suite=SUITE1, SUITE2, ..., SUITEn
36
37 Interpreting test results
38 =========================
39@@ -239,20 +240,24 @@
40
41 .. program:: kewpie.py
42
43+.. option:: --version
44+
45+ show program's version number and exit
46+
47 .. option:: -h, --help
48
49 show this help message and exit
50
51 Configuration controls - kewpie can read config files with certain options pre-set:
52----------------------------------------------------------------------------------------------------
53+-----------------------------------------------------------------------------------
54
55 .. option:: --sys_config_file=SYSCONFIGFILEPATH
56
57 The file that specifies system configuration specs for
58 kewpie to execute tests (not yet implemented)
59
60-Options for the test-runner itself
61-----------------------------------
62+Options for the test-runner itself - defining the system under test and how to execute tests:
63+---------------------------------------------------------------------------------------------
64
65 .. program:: kewpie.py
66
67@@ -293,6 +298,19 @@
68 (currently just a placeholder) [False]
69
70
71+Options for controlling how tests are executed
72+----------------------------------------------
73+
74+.. program:: kewpie.py
75+
76+.. option:: --test-debug
77+
78+ Toggle to control any debugging / helper output with unittest test cases [False]
79+
80+.. option:: --randgen-seed=RANDGENSEED
81+
82+ Alter the seed value provided to the random query generator to vary test runs. (string) [1]
83+
84 Options for controlling which tests are executed
85 ------------------------------------------------
86
87@@ -331,6 +349,10 @@
88 a given sequence, the first test will be run n times,
89 then the second, etc [1]
90
91+.. option:: --email-report-tgt=EMAILREPORTTGT
92+
93+ Used to send report mails. Sends the report to the specified email-ID
94+
95 Options for defining the code that will be under test
96 -----------------------------------------------------
97
98@@ -344,6 +366,11 @@
99 relative to the argument (client-bindir,
100 serverdir, testdir) [../]
101
102+.. option:: --default-server-type=DEFAULTSERVERTYPE
103+
104+ Defines what we consider to be the default server type.
105+ We assume a server is default type unless specified otherwise [drizzle]
106+
107 .. option:: --serverdir=SERVERPATH
108
109 Path to the server executable. [auto-search]
110@@ -420,16 +447,29 @@
111
112 The path the xtrabackup binary to be tested
113
114+.. option:: --tar4ibd-path=TAR4IBDPATH
115+
116+ The path to the tar4ibd binary that will be used for any applicable tests
117+
118 .. option:: --wsrep-provider-path=WSREPPROVIDER
119
120 The path to a wsrep provider library for use with
121 mysql
122+
123+.. option:: --cluster-cnf=CLUSTERCNF
124+
125+ The path to a config file defining a running cluster (node info)
126
127 .. option:: --subunit-outfile=SUBUNITOUTFILE
128
129 File path where subunit output will be logged
130 [/kewpie/workdir/test_results.subunit]
131
132+.. option:: --results-db-dsn=RESULTSDBDSN
133+
134+ Specifies the database connection Default string:
135+ 127.0.0.1:root::results_db:3306
136+
137 Options to pass options on to the server
138 -----------------------------------------
139
140
141=== modified file 'docs/testing/sysbench.rst'
142--- docs/testing/sysbench.rst 2012-03-08 00:54:48 +0000
143+++ docs/testing/sysbench.rst 2012-08-21 17:59:20 +0000
144@@ -24,23 +24,50 @@
145 $> cd drizzle-sysbench
146 $> ./autogen.sh && ./configure && make && sudo make install
147
148-Make sure sysbench is then in your path
149+Make sure sysbench is in your path
150
151
152 sysbench / kewpie tests
153 =======================
154
155-A sysbench test defines a run for a particular concurrency. There are suites for readonly and readwrite.
156-They are currently broken down this way as an experiment - we are open to other ways of organizing these tests::
157-
158- [test_info]
159- comment = 16 threads
160-
161- [test_command]
162- command = sysbench --max-time=240 --max-requests=0 --test=oltp --db-ps-mode=disable --drizzle-table-engine=innodb --oltp-read-only=on --oltp-table-size=1000000 --drizzle-mysql=on --drizzle-user=root --drizzle-db=test --drizzle-port=$MASTER_MYPORT --drizzle-host=localhost --db-driver=drizzle --num-threads=16
163-
164- [test_servers]
165- servers = [[innodb.buffer-pool-size=256M innodb.log-file-size=64M innodb.log-buffer-size=8M innodb.thread-concurrency=0 innodb.additional-mem-pool-size=16M table-open-cache=4096 table-definition-cache=4096 mysql-protocol.max-connections=2048]]
166+The sysbench test suite consists of python unittests that encapsulate / automate sysbench oltp runs for readonly and readwrite.
167+
168+The tests are written in Python and are rather straightforward to modify.
169+
170+Changing server options:
171+
172+::
173+
174+ #drizzle options
175+ server_requirements = [['innodb.buffer-pool-size=256M innodb.log-file-size=64M innodb.log-buffer-size=8M innodb.thread-concurrency=0 innodb.additional-mem-pool-size=16M table-open-cache=4096 table-definition-cache=4096 mysql-protocol.max-connections=2048']]
176+
177+ #mysql options
178+ server_requirements = [['innodb_buffer_pool_size=256M innodb_log_file_size=64M innodb_log_buffer_size=8M innodb_thread_concurrency=0 innodb_additional_mem_pool_size=16M table_open_cache=4096 table_definition_cache=4096 max_connections=2048']]
179+
180+
181+Altering concurrencies tested:
182+
183+The concurrencies for the tests are specified in the /tests/lib/util/sysbenchTestCase.py file. The concurrencies can be changed by directly editing them in the file mentioned. ( The tests are straightforward to modify / flexible )
184+
185+::
186+
187+ #various concurrencies to use with sysbench
188+ self.concurrencies = [ 16, 32, 64, 128, 256, 512, 1024 ]
189+
190+This line can be modified according to the need
191+
192+
193+Altering iterations:
194+
195+By default, the tests will execute 3 iterations of each concurrency tested. This is to help achieve some consistency for performance analysis. However, the number of iterations can be varied, by editing them in the file /tests/lib/util/sysbenchTestCase.py
196+
197+::
198+
199+ #how many times to run sysbench at each concurrency
200+ self.iterations = 3
201+
202+This line can be modified according to the need
203+
204
205 Running tests
206 =============
207@@ -52,72 +79,98 @@
208 :option:`kewpie.py --force` is recommended if you are running several tests
209 - it will allow you to view all successes and failures in one run.
210
211-Running individual tests
212+Running tests
213 ------------------------
214 If one only wants to run a few, specific tests, they may do so this way::
215
216- ./kewpie --mode=sysbench [OPTIONS] test1 [test2 ... testN]
217-
218-Running all tests within a suite
219---------------------------------
220-Many of the tests supplied with Drizzle are organized into suites.
221-
222-The tests within drizzle/tests/randgen_tests/main are considered the 'main' suite.
223-Other suites are also subdirectories of drizzle/tests/randgen_tests.
224-
225-To run the tests in a specific suite::
226-
227- ./kewpie --mode=sysbench [OPTIONS] --suite=SUITENAME
228-
229-Running specific tests within a suite
230---------------------------------------
231-To run a specific set of tests within a suite::
232-
233- ./kewpie --mode=sysbench [OPTIONS] --suite=SUITENAME TEST1 [TEST2..TESTN]
234-
235-Calling tests using <suitename>.<testname> currently does not work. One must
236-specify the test suite via the :option:`kewpie.py --suite` option.
237-
238-
239-Running all available tests
240----------------------------
241-One would currently have to name all suites, but the majority of the working tests live in the main suite
242-Other suites utilize more exotic server combinations and we are currently tweaking them to better integrate with the
243-kewpie system. The slave-plugin suite does currently have a good config file for setting up simple replication setups for testing.
244-To execute several suites' worth of tests::
245-
246- ./kewpie --mode=sysbench [OPTIONS] --suite=SUITE1, SUITE2, ...SUITEN
247-
248-Interpreting test results
249-=========================
250-The output of the test runner is quite simple. Every test should pass.
251-In the event of a test failure, please take the time to file a bug here:
252-*https://bugs.launchpad.net/drizzle*
253-
254-During a run, the program will provide the user with:
255- * test name (suite + name)
256- * test status (pass/fail/skipped)
257- * time spent executing each test
258-
259-Example output::
260-
261- 20110601-191706 ===============================================================
262- 20110601-191706 TEST NAME [ RESULT ] TIME (ms)
263- 20110601-191706 ===============================================================
264- 20110601-191706 readonly.concurrency_16 [ pass ] 240019
265- 20110601-191706 max_req_lat_ms: 21.44
266- 20110601-191706 rwreqps: 4208.2
267- 20110601-191706 min_req_lat_ms: 6.31
268- 20110601-191706 deadlocksps: 0.0
269- 20110601-191706 tps: 150.29
270- 20110601-191706 avg_req_lat_ms: 6.65
271- 20110601-191706 95p_req_lat_ms: 7.02
272- 20110601-191706 ===============================================================
273- 20110601-191706 INFO Test execution complete in 275 seconds
274- 20110601-191706 INFO Summary report:
275- 20110601-191706 INFO Executed 1/1 test cases, 100.00 percent
276- 20110601-191706 INFO STATUS: PASS, 1/1 test cases, 100.00 percent executed
277- 20110601-191706 INFO Spent 240 / 275 seconds on: TEST(s)
278- 20110601-191706 INFO Test execution complete
279- 20110601-191706 INFO Stopping all running servers...
280+ ./kewpie --suite=sysbench [OPTIONS] sysbench_readonly_test | sysbench_readwrite_test
281+
282+.. note:: Calling tests using <suitename>.<testname> currently does not work. One must specify the test suite via the :option:`kewpie.py --suite` option.
283+
284+Results database
285+------------------
286+The Drizzle team has ported drizzle-automation functionality for sysbench.
287+This means that a user can run a Drizzle / MySQL database and use it to
288+store sysbench run data and to compare runs against historic data.
289+
290+The SQL to create the required tables are in tests/std_data/sysbench_db.sql
291+
292+This will create 3 tables:
293+
294+ * bench_config
295+ * bench_runs
296+ * sysbench_run_iterations
297+
298+This emulates drizzle-automation's historic behavior.
299+
300+Once the tables have been created, add the following option to the kewpie call:
301+--results-db-dsn='host_ip:user:user_pass:drizzle_stats:port'
302+
303+
304+Reporting test result
305+---------------------
306+Once the benchmark test(s) get(s) completed, a report is generated.
307+
308+Given below is an example of the test report
309+
310+::
311+
312+ ====================================================================================================
313+ SYSBENCH BENCHMARK REPORT
314+ ====================================================================================================
315+ MACHINE: erlking
316+ RUN ID: 19
317+ RUN DATE: 2012-08-15T18:16:12.596937
318+ WORKLOAD: sysbench
319+ SERVER: drizzle
320+ VERSION: staging
321+ REVISION: 2595
322+ COMMENT: 2595: Ported this reporting capability!
323+ ====================================================================================================
324+
325+ TRENDING OVER LAST 5 runs
326+ Conc TPS % Diff from Avg Diff Min Max Avg STD
327+ ====================================================================================================
328+ 128 218.72 +0.58% 1.26 192.64 232.90 217.46 13.02
329+ 256 223.71 +6.85% 14.34 192.28 227.73 209.37 14.07
330+ 512 223.40 +7.47% 15.53 192.70 228.93 207.87 12.76
331+ ====================================================================================================
332+
333+ TRENDING OVER Last 20 runs
334+
335+ Conc TPS % Diff from Avg Diff Min Max Avg STD
336+ ====================================================================================================
337+ 128 218.72 +2.95% 6.26 188.42 232.90 212.45 15.27
338+ 256 223.71 +7.92% 16.42 189.18 232.99 207.29 14.94
339+ 512 223.40 +8.46% 17.42 191.35 232.79 205.98 14.09
340+ ====================================================================================================
341+
342+ TRENDING OVER ALL runs
343+
344+ Conc TPS % Diff from Avg Diff Min Max Avg STD
345+ ====================================================================================================
346+ 128 218.72 +2.95% 6.26 188.42 232.90 212.45 15.27
347+ 256 223.71 +7.92% 16.42 189.18 232.99 207.29 14.94
348+ 512 223.40 +8.46% 17.42 191.35 232.79 205.98 14.09
349+ ====================================================================================================
350+ 20120815-184028 sysbench.sysbench_readonly_test [ pass ] 1455879
351+ 20120815-184028 ===============================================================
352+
353+Email test report
354+-----------------
355+
356+Another drizzle-automation functionality that is ported to kewpie's sysbench is emailing test report to the specified email ID.
357+
358+::
359+
360+ ./kewpie --suite=sysbench --email-report-tgt=foo@bar.com
361+
362+The output, after including the mailing option:
363+
364+::
365+
366+ 20120815-184028 Mailing report...
367+ 20120815-184028 To: foo@bar.com
368+ 20120815-184028 From: smtplibpython@gmail.com
369+ 20120815-184028 Report successfully sent...
370
371
372=== modified file 'tests/lib/modes/native/native_test_execution.py'
373--- tests/lib/modes/native/native_test_execution.py 2011-11-28 21:21:03 +0000
374+++ tests/lib/modes/native/native_test_execution.py 2012-08-21 17:59:20 +0000
375@@ -70,6 +70,8 @@
376 test_module.servers = self.current_servers
377 test_module.test_executor = self
378 test_module.server_manager = self.server_manager
379+ test_module.mail_tgt = self.system_manager.variables['emailreporttgt']
380+ test_module.dsn_string = self.system_manager.variables['resultsdbdsn']
381
382 # start our test
383 self.time_manager.start(testcase_name,'test')
384
385=== modified file 'tests/lib/opts/test_run_options.py'
386--- tests/lib/opts/test_run_options.py 2012-05-25 18:41:11 +0000
387+++ tests/lib/opts/test_run_options.py 2012-08-21 17:59:20 +0000
388@@ -276,6 +276,16 @@
389 , help = "Run each test case the specified number of times. For a given sequence, the first test will be run n times, then the second, etc [%default]"
390 )
391
392+
393+ test_control_group.add_option(
394+ "--email-report-tgt"
395+ , dest="emailreporttgt"
396+ , action="store"
397+ , type='string'
398+ , default=None
399+ , help="Used to send report mails. sends the test report to the email ID given"
400+ )
401+
402 parser.add_option_group(test_control_group)
403
404 # test subject control group
405@@ -469,6 +479,14 @@
406 , help = "File path where subunit output will be logged [%default]"
407 )
408
409+ environment_control_group.add_option(
410+ "--results-db-dsn"
411+ , dest="resultsdbdsn"
412+ , action='store'
413+ , default=None
414+ , help = "Specifies the database connection\nDefault string:'127.0.0.1:root::results_db:3306'"
415+ )
416+
417 parser.add_option_group(environment_control_group)
418 # end environment control group
419
420
421=== modified file 'tests/lib/server_mgmt/drizzled.py'
422--- tests/lib/server_mgmt/drizzled.py 2012-06-19 18:46:42 +0000
423+++ tests/lib/server_mgmt/drizzled.py 2012-08-21 17:59:20 +0000
424@@ -200,8 +200,9 @@
425 # This is what test-run.pl does and it helps us pass logging_stats tests
426 # while not self.ping_server(server, quiet=True) and timer != timeout:
427
428- return self.system_manager.find_path( [self.pid_file]
429- , required=0)
430+ #return self.system_manager.find_path( [self.pid_file]
431+ # , required=0)
432+ return self.ping(quiet=True)
433
434 def create_slave_config_file(self):
435 """ Create a config file suitable for use
436
437=== modified file 'tests/lib/server_mgmt/server.py'
438--- tests/lib/server_mgmt/server.py 2012-02-09 02:28:38 +0000
439+++ tests/lib/server_mgmt/server.py 2012-08-21 17:59:20 +0000
440@@ -29,6 +29,7 @@
441 # imports
442 import os
443 import time
444+import commands
445 import subprocess
446
447 from lib.util.mysql_methods import execute_query
448@@ -379,4 +380,29 @@
449 with open(self.error_log,'r') as errlog:
450 data = errlog.readlines()
451 return ''.join(data)
452+
453+ def get_bzr_info(self):
454+ """ Find the revision comment from BZR.
455+ Taken from drizzle-automation's codebase
456+
457+ """
458+ os.chdir(self.code_tree.basedir)
459+ (retcode, rev_comment_output)= commands.getstatusoutput("bzr log -r-1 -n0 --line")
460+
461+ # Output from above command looks like this:
462+ # jpipes@serialcoder:~/repos/drizzle/trunk-sysbench-r1046$ bzr log -r-1 -n0 --line
463+ # 1046: Brian Aker 2009-05-31 [merge] Merge Jay.
464+ # 1039.2.9: Jay Pipes 2009-05-31 Tiny cleanups
465+ # 1039.2.8: Jay Pipes 2009-05-31 Yet more indentation and style cleanup
466+ # 1039.2.7: Jay Pipes 2009-05-31 Yet more style and indentation cleanups.
467+ # 1039.2.6: Jay Pipes 2009-05-31 No code changes...only indentation and style cleanup.
468+
469+ comment_lines= rev_comment_output.split("\n")
470+ comment_lines = [line for line in comment_lines if line != 'cannot import name info']
471+ rev_comment= comment_lines[0]
472+ if len(comment_lines) > 1:
473+ full_commentary= "\n".join(comment_lines[1:])
474+ else:
475+ full_commentary= None
476+ return rev_comment.split(':')[0], rev_comment
477
478
479=== modified file 'tests/lib/sys_mgmt/system_management.py'
480--- tests/lib/sys_mgmt/system_management.py 2012-02-09 02:28:38 +0000
481+++ tests/lib/sys_mgmt/system_management.py 2012-08-21 17:59:20 +0000
482@@ -104,6 +104,7 @@
483 ))
484
485 self.wsrep_provider_path = variables['wsrepprovider']
486+ self.variables = variables
487
488 # we use this to preface commands in order to run valgrind and such
489 self.cmd_prefix = ''
490
491=== modified file 'tests/lib/util/crashme_methods.py'
492--- tests/lib/util/crashme_methods.py 2012-06-12 23:02:14 +0000
493+++ tests/lib/util/crashme_methods.py 2012-08-21 17:59:20 +0000
494@@ -21,6 +21,7 @@
495
496 import os
497 import subprocess
498+import re
499
500 """ crashme_methods
501
502@@ -83,6 +84,12 @@
503 output = ''.join(crashme_file.readlines())
504 bot.logging.debug(output)
505 crashme_file.close()
506+
507+ print output
508+ regex={'order':re.compile(r".*number.*")}
509+ for line in output.split("\n"):
510+ report=regex['order'].match(line)
511+ print "\nmatching regex...\n",report
512
513 bot.logging.debug("crashme_retcode: %d" %(retcode))
514 bot.current_test_retcode = retcode
515
516=== added file 'tests/lib/util/database_connect.py'
517--- tests/lib/util/database_connect.py 1970-01-01 00:00:00 +0000
518+++ tests/lib/util/database_connect.py 2012-08-21 17:59:20 +0000
519@@ -0,0 +1,62 @@
520+import MySQLdb
521+
522+def results_db_fetch(cursor,query):
523+ """used to fetch data from database
524+ and return a dictionary containing the required values
525+
526+ """
527+
528+ cursor.execute(query)
529+ data=cursor.fetchone()
530+ fetch={
531+ 'tps':data[1],
532+ 'min_req_lat_ms':data[2],
533+ 'max_req_lat_ms':data[3],
534+ 'avg_req_lat_ms':data[4],
535+ '95p_req_lat_ms':data[5],
536+ 'rwreqps':data[6],
537+ 'deadlocksps':data[7]
538+ }
539+ return fetch
540+
541+
542+def results_db_connect(dsn_string,operation,query):
543+ """used to establish a database connection
544+
545+ """
546+
547+ #getting the connection parameters
548+ connect_param=dsn_string.split(":")
549+
550+ #establishing the connection
551+ connection=MySQLdb.connect(host=connect_param[0],user=connect_param[1],passwd=connect_param[2],db=connect_param[3],port=int(connect_param[4]))
552+ cursor=connection.cursor()
553+ sql=query
554+
555+ #select operation - selects tests results from database and returns a dictionary
556+ if operation=="select":
557+
558+ # returns a fetch value for the concurrency-iteration value which is not yet recorded
559+ try:
560+ fetch=results_db_fetch(cursor,query)
561+ except TypeError:
562+ fetch={
563+ 'tps':0.0,
564+ 'min_req_lat_ms':0.0,
565+ 'max_req_lat_ms':0.0,
566+ 'avg_req_lat_ms':0.0,
567+ '95p_req_lat_ms':0.0,
568+ 'rwreqps':0.0,
569+ 'deadlocksps':0.0
570+ }
571+ cursor.close()
572+ connection.close()
573+ return fetch
574+
575+ #update operation - updates the database with the new test result
576+ elif operation=="insert" or operation=="delete":
577+ cursor.execute(query)
578+ connection.autocommit(True)
579+ cursor.close()
580+ connection.close()
581+
582
583=== added file 'tests/lib/util/drizzleslap_methods.py'
584--- tests/lib/util/drizzleslap_methods.py 1970-01-01 00:00:00 +0000
585+++ tests/lib/util/drizzleslap_methods.py 2012-08-21 17:59:20 +0000
586@@ -0,0 +1,113 @@
587+#! /usr/bin/env python
588+# -*- mode: python; indent-tabs-mode: nil; -*-
589+# vim:expandtab:shiftwidth=2:tabstop=2:smarttab:
590+#
591+# Copyright (C) 2011 Patrick Crews
592+# Copyright (C) 2012 M.Sharan Kumar
593+#
594+#
595+# This program is free software; you can redistribute it and/or modify
596+# it under the terms of the GNU General Public License as published by
597+# the Free Software Foundation; either version 2 of the License, or
598+# (at your option) any later version.
599+#
600+# This program is distributed in the hope that it will be useful,
601+# but WITHOUT ANY WARRANTY; without even the implied warranty of
602+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
603+# GNU General Public License for more details.
604+#
605+# You should have received a copy of the GNU General Public License
606+# along with this program; if not, write to the Free Software
607+# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
608+
609+import os
610+import re
611+import subprocess
612+
613+def prepare_drizzleslap(test_executor, test_cmd):
614+ """ Prepare the server for a drizzleslap test run
615+
616+ """
617+ bot = test_executor
618+ drizzleslap_outfile = os.path.join(bot.logdir,'drizzleslap.out')
619+ drizzleslap_output = open(drizzleslap_outfile,'w')
620+ drizzleslap_prep_cmd = ' '.join([test_cmd,'prepare'])
621+ bot.logging.info("Preparing database for drizzleslap run...")
622+ bot.logging.verbose(drizzleslap_prep_cmd)
623+ drizzleslap_subproc = subprocess.Popen( drizzleslap_prep_cmd
624+ , shell = True
625+ , env = bot.working_environment
626+ , stdout = sysbench_output
627+ , stderr = subprocess.STDOUT
628+ )
629+ drizzleslap_subproc.wait()
630+ retcode = drizzleslap_subproc.returncode
631+ drizzleslap_output.close()
632+ with open(drizzleslap_outfile,'r') as drizzleslap_file:
633+ output = ''.join(drizzleslap_file.readlines())
634+ drizzleslap_file.close()
635+ bot.logging.verbose("drizzleslap_retcode: %d" %(retcode))
636+ return retcode, output
637+
638+def execute_drizzleslap(test_executor, test_cmd):
639+ """ Execute the commandline and return the result.
640+ We use subprocess as we can pass os.environ dicts and whatnot
641+
642+ """
643+
644+ bot = test_executor
645+ drizzleslap_cmd = ' '.join([test_cmd, 'run'])
646+ bot.logging.info("Executing : %s" %(drizzleslap_cmd))
647+ drizzleslap_outfile = os.path.join(bot.logdir,'drizzleslap.out')
648+ with open(drizzleslap_outfile,'w') as drizzleslap_output:
649+ drizzleslap_subproc = subprocess.Popen( drizzleslap_cmd
650+ , shell = True
651+ , env = bot.working_environment
652+ , stdout = sysbench_output
653+ , stderr = subprocess.STDOUT
654+ )
655+ drizzleslap_subproc.wait()
656+ drizzleslap_output.close()
657+ retcode = drizzleslap_subproc.returncode
658+
659+ drizzleslap_file = open(drizzlslap_outfile,'r')
660+ output = ''.join(drizzleslap_file.readlines())
661+ bot.logging.debug(output)
662+ drizzleslap_file.close()
663+ return retcode, output
664+
665+def process_drizzleslap_output(test_output):
666+ """ drizzleslap has run, we now check out what we have
667+ We also output the data from the run
668+
669+ """
670+ # This slice code taken from drizzle-automation's drizzleslap handling
671+ # Slice up the output report into a matrix and insert into the DB.
672+
673+
674+ #TODO complete the regexes
675+ regexes= {
676+ 'run_id': re.compile()
677+ , 'engine_name': re.compile()
678+ , 'test_name': re.compile()
679+ , 'queries_avg': re.compile()
680+ , 'queries_min': re.compile()
681+ , 'queries_max': re.compile()
682+ , 'total_time': re.compile()
683+ , 'stddev':re.compile()
684+ , 'iterations':re.compile()
685+ , 'concurrency':re.compile()
686+ , 'concurrency2':re.compile()
687+ , 'queries_per_client':re.compile()
688+ }
689+ run= {}
690+ for line in test_output.split("\n"):
691+ for key in regexes.keys():
692+ result= regexes[key].match(line)
693+ if result:
694+ run[key]= float(result.group(1)) # group(0) is entire match...
695+ # we set our test output to the regex'd-up data
696+ # we also make it a single string, separated by newlines
697+ parsed_test_output = str(run)[1:-1].replace(',','\n').replace("'",'')
698+ return parsed_test_output
699+
700
701=== added file 'tests/lib/util/mailing_report.py'
702--- tests/lib/util/mailing_report.py 1970-01-01 00:00:00 +0000
703+++ tests/lib/util/mailing_report.py 2012-08-21 17:59:20 +0000
704@@ -0,0 +1,44 @@
705+#! /usr/bin/env python
706+# -*- mode: python; indent-tabs-mode: nil; -*-
707+# vim:expandtab:shiftwidth=2:tabstop=2:smarttab:
708+#
709+# Copyright (C) 2012 M.Sharan Kumar
710+#
711+#
712+# This program is free software; you can redistribute it and/or modify
713+# it under the terms of the GNU General Public License as published by
714+# the Free Software Foundation; either version 2 of the License, or
715+# (at your option) any later version.
716+#
717+# This program is distributed in the hope that it will be useful,
718+# but WITHOUT ANY WARRANTY; without even the implied warranty of
719+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
720+# GNU General Public License for more details.
721+#
722+# You should have received a copy of the GNU General Public License
723+# along with this program; if not, write to the Free Software
724+# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
725+
726+import smtplib
727+
728+def sendMail(mail_executor,to_address,message):
729+ """mails the report of sysbench test"""
730+
731+ logging=mail_executor.logging
732+
733+ #configuring sender's login
734+ from_address='smtplibpython@gmail.com'
735+ from_password='smtpprotocol'
736+
737+ #sending mail to specified to_addresses
738+ logging.info("Mailing report...")
739+ logging.info("To: %s" %to_address)
740+ logging.info("From: %s" %from_address)
741+ server=smtplib.SMTP('smtp.gmail.com',587)
742+ server.ehlo()
743+ server.starttls()
744+ server.ehlo()
745+ server.login(from_address,from_password)
746+ server.sendmail(from_address,to_address,message)
747+ logging.info("Report successfully sent...")
748+ server.close()
749
750=== modified file 'tests/lib/util/mysql_methods.py'
751--- tests/lib/util/mysql_methods.py 2011-12-01 18:41:42 +0000
752+++ tests/lib/util/mysql_methods.py 2012-08-21 17:59:20 +0000
753@@ -207,14 +207,24 @@
754 return return_data
755
756 def execute_query( query
757- , server
758+ , server = None
759 , server_host = '127.0.0.1'
760- , schema='test'):
761+ , schema='test'
762+ , dsn_string = None):
763 try:
764- conn = MySQLdb.connect( host = server_host
765- , port = server.master_port
766- , user = 'root'
767- , db = schema)
768+ if dsn_string:
769+ #getting the connection parameters from dsn_string
770+ connect_param=dsn_string.split(":")
771+ conn = MySQLdb.connect( host=connect_param[0]
772+ , user=connect_param[1]
773+ , passwd=connect_param[2]
774+ , db=connect_param[3]
775+ , port=int(connect_param[4]))
776+ else:
777+ conn = MySQLdb.connect( host = server_host
778+ , port = server.master_port
779+ , user = 'root'
780+ , db = schema)
781 cursor = conn.cursor()
782 cursor.execute(query)
783 result_set = cursor.fetchall()
784
785=== added file 'tests/lib/util/sysbenchTestCase.py'
786--- tests/lib/util/sysbenchTestCase.py 1970-01-01 00:00:00 +0000
787+++ tests/lib/util/sysbenchTestCase.py 2012-08-21 17:59:20 +0000
788@@ -0,0 +1,156 @@
789+#! /usr/bin/env python
790+# -*- mode: python; indent-tabs-mode: nil; -*-
791+# vim:expandtab:shiftwidth=2:tabstop=2:smarttab:
792+#
793+# Copyright (C) 2012 Patrick Crews, M.Sharan Kumar
794+#
795+#
796+# This program is free software; you can redistribute it and/or modify
797+# it under the terms of the GNU General Public License as published by
798+# the Free Software Foundation; either version 2 of the License, or
799+# (at your option) any later version.
800+#
801+# This program is distributed in the hope that it will be useful,
802+# but WITHOUT ANY WARRANTY; without even the implied warranty of
803+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
804+# GNU General Public License for more details.
805+#
806+# You should have received a copy of the GNU General Public License
807+# along with this program; if not, write to the Free Software
808+# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
809+
810+import re
811+import time
812+import socket
813+import subprocess
814+import datetime
815+from copy import deepcopy
816+
817+from lib.util.sysbench_methods import prepare_sysbench
818+from lib.util.sysbench_methods import execute_sysbench
819+from lib.util.sysbench_methods import process_sysbench_output
820+from lib.util.sysbench_methods import sysbench_db_analysis
821+from lib.util.sysbench_methods import getSysbenchReport
822+from lib.util.mysqlBaseTestCase import mysqlBaseTestCase
823+from lib.util.database_connect import results_db_connect
824+from lib.util.mailing_report import sendMail
825+
826+servers = []
827+server_manager = None
828+test_executor = None
829+
830+class sysbenchTestCase(mysqlBaseTestCase):
831+
832+ # initializing test_data ( data for regression analysis )
833+ def initTestData(self,test_executor,servers):
834+ self.test_executor = test_executor
835+ self.logging = test_executor.logging
836+ self.servers = servers
837+ self.master_server = servers[0]
838+ self.test_data = {}
839+ self.test_cmd = []
840+ self.iterations = 0
841+ self.concurrencies = []
842+
843+ # data for results database / regression analysis
844+ self.test_data['run_date']= datetime.datetime.now().isoformat()
845+ self.test_data['test_machine'] = socket.gethostname()
846+ self.test_data['test_server_type'] = self.master_server.type
847+ self.test_data['test_server_revno'], self.test_data['test_server_comment'] = self.master_server.get_bzr_info()
848+ self.test_data['config_name'] = self.config_name
849+
850+
851+ # utility code for configuring and preparing sysbench test
852+ def prepareSysbench(self,test_cmd,test_executor,servers):
853+
854+ # creating the initial test data
855+ self.initTestData(test_executor,servers)
856+ # creating the initial test command
857+ self.test_cmd = test_cmd
858+ # how many times to run sysbench at each concurrency
859+ self.iterations = 3
860+ # various concurrencies to use with sysbench
861+ # self.concurrencies = [16, 32, 64, 128, 256, 512, 1024 ]
862+ self.concurrencies = [ 128, 256, 512 ]
863+
864+ # we setup once. This is a readonly test and we don't
865+ # alter the test bed once it is created
866+ exec_cmd = " ".join(self.test_cmd)
867+ retcode, output = prepare_sysbench(test_executor, exec_cmd)
868+ err_msg = ("sysbench 'prepare' phase failed.\n"
869+ "retcode: %d"
870+ "output: %s" %(retcode,output))
871+ self.assertEqual(retcode, 0, msg = err_msg)
872+
873+
874+ # the __main__ code for executing sysbench oltp test
875+ def executeSysbench(self):
876+
877+ # executing sysbench test
878+ for concurrency in self.concurrencies:
879+ if concurrency not in self.test_data:
880+ self.test_data[concurrency] = []
881+ exec_cmd = " ".join(self.test_cmd)
882+ exec_cmd += " --num-threads=%d" % concurrency
883+ for test_iteration in range(self.iterations):
884+ self.logging.info("Concurrency: %d Iteration: %d" % (concurrency, test_iteration+1) )
885+ retcode, output = execute_sysbench(self.test_executor, exec_cmd)
886+ self.assertEqual(retcode, 0, msg = output)
887+ # This might be inefficient/redundant...perhaps remove later
888+ parsed_output = process_sysbench_output(output)
889+ self.logging.info(parsed_output)
890+
891+ # gathering the data from the output
892+ self.saveTestData(test_iteration,concurrency,output)
893+
894+ # utility code for saving test run information for given concurrency
895+ def saveTestData(self,test_iteration,concurrency,output):
896+
897+ # creating regexes for test result
898+ regexes={ 'tps':re.compile(r".*transactions\:\s+\d+\D*(\d+\.\d+).*")
899+ , 'min_req_lat_ms':re.compile(r".*min\:\s+(\d*\.\d+)ms.*")
900+ , 'max_req_lat_ms':re.compile(r".*max\:\s+(\d*\.\d+)ms.*")
901+ , 'avg_req_lat_ms':re.compile(r".*avg\:\s+(\d*\.\d+)ms.*")
902+ , '95p_req_lat_ms':re.compile(r".*approx.\s+95\s+percentile\:\s+(\d+\.\d+)ms.*")
903+ , 'rwreqps':re.compile(r".*read\/write\s+requests\:\s+\d+\D*(\d+\.\d+).*")
904+ , 'deadlocksps':re.compile(r".*deadlocks\:\s+\d+\D*(\d+\.\d+).*")
905+ }
906+
907+ run={}
908+ for line in output.split("\n"):
909+ for key in regexes:
910+ result=regexes[key].match(line)
911+ if result:
912+ run[key]=float(result.group(1))
913+ run['mode']="readonly"
914+ run['iteration'] = test_iteration
915+ self.test_data[concurrency].append(deepcopy(run))
916+ #return self.test_data
917+
918+ # If provided with a results_db, we process our data
919+ # utility code for reporting the test data
920+ def reportTestData(self,dsn_string,mail_tgt):
921+
922+ # Report data
923+ msg_data = []
924+ test_concurrencies = [i for i in self.test_data.keys() if type(i) is int]
925+ test_concurrencies.sort()
926+ for concurrency in test_concurrencies:
927+ msg_data.append('Concurrency: %s' %concurrency)
928+ for test_iteration in self.test_data[concurrency]:
929+ msg_data.append("Iteration: %s || TPS: %s" %(test_iteration['iteration']+1, test_iteration['tps']))
930+ for line in msg_data:
931+ self.logging.info(line)
932+
933+ # Store / analyze data in results db, if available
934+ if dsn_string:
935+ result, msg_data = sysbench_db_analysis(dsn_string, self.test_data)
936+ self.logging.info(msg_data)
937+
938+ # mailing sysbench report
939+ if mail_tgt:
940+ sendMail(test_executor,mail_tgt,"\n".join(msg_data))
941+
942+ def tearDown(self):
943+ server_manager.reset_servers(test_executor.name)
944+
945
946=== modified file 'tests/lib/util/sysbench_methods.py'
947--- tests/lib/util/sysbench_methods.py 2012-06-19 18:58:23 +0000
948+++ tests/lib/util/sysbench_methods.py 2012-08-21 17:59:20 +0000
949@@ -2,7 +2,7 @@
950 # -*- mode: python; indent-tabs-mode: nil; -*-
951 # vim:expandtab:shiftwidth=2:tabstop=2:smarttab:
952 #
953-# Copyright (C) 2011 Patrick Crews
954+# Copyright (C) 2011, 2012 Patrick Crews, M.Sharan Kumar
955 #
956 #
957 # This program is free software; you can redistribute it and/or modify
958@@ -23,6 +23,8 @@
959 import re
960 import subprocess
961
962+from lib.util.mysql_methods import execute_query
963+
964 def prepare_sysbench(test_executor, test_cmd):
965 """ Prepare the server for a sysbench run
966
967@@ -100,5 +102,350 @@
968 # we set our test output to the regex'd-up data
969 # we also make it a single string, separated by newlines
970 parsed_test_output = str(run)[1:-1].replace(',','\n').replace("'",'')
971- return parsed_test_output
972+ return parsed_test_output
973+
974+def sysbench_db_analysis(dsn_string, test_data):
975+ """ Process the data from a sysbench run by
976+ INSERTing the data into the provided dsn_string / database
977+ and doing checks on performance regressions (against tps)
978+
979+ test_data is a dictionary whose keys are the concurrencies
980+ used in the test and whose data is a list of dictionaries,
981+ one per iteration taken for said concurrency
982+
983+ """
984+
985+ # log our sysbench_run
986+ run_id = getNextRunId(dsn_string)
987+ config_id = getConfigId(dsn_string, test_data)
988+ log_sysbench_run( run_id
989+ , config_id
990+ , test_data['test_machine']
991+ , "staging-%s" %test_data['test_server_revno']
992+ , test_data['run_date']
993+ , dsn_string
994+ )
995+ test_concurrencies = [ i for i in test_data.keys() if type(i) is int ]
996+ test_concurrencies.sort()
997+ for concurrency in test_concurrencies:
998+ for iteration_data in test_data[concurrency]:
999+ log_sysbench_iteration(run_id, concurrency, iteration_data, dsn_string)
1000+ msg_data = getSysbenchRegressionReport(run_id, test_data, dsn_string)
1001+ return 0, msg_data
1002+
1003+def log_sysbench_run(run_id, config_id, server_name, server_version, run_date, dsn_string):
1004+ """Creates a new run record in the database for this run"""
1005+
1006+ query = """INSERT INTO bench_runs (
1007+ run_id
1008+ , config_id
1009+ , server
1010+ , version
1011+ , run_date
1012+ ) VALUES (%d, %d, '%s', '%s', '%s')
1013+ """ % ( run_id
1014+ , config_id
1015+ , server_name
1016+ , server_version
1017+ , run_date
1018+ )
1019+ retcode, result= execute_query(query, dsn_string=dsn_string)
1020+ return result
1021+
1022+
1023+def log_sysbench_iteration(run_id, concurrency, iteration_data, dsn_string):
1024+ # TODO: make sure we properly capture full commentary
1025+ full_commentary = None
1026+ # Write results to the DB
1027+ query = """INSERT INTO sysbench_run_iterations (
1028+ run_id
1029+ , concurrency
1030+ , iteration
1031+ , tps
1032+ , read_write_req_per_second
1033+ , deadlocks_per_second
1034+ , min_req_latency_ms
1035+ , max_req_latency_ms
1036+ , avg_req_latency_ms
1037+ , 95p_req_latency_ms
1038+ ) VALUES (%d, %d, %d, %0.2f, %0.2f, %0.2f, %0.2f, %0.2f, %0.2f, %0.2f)
1039+ """ % ( int(run_id)
1040+ , int(concurrency)
1041+ , int(iteration_data['iteration'])
1042+ , iteration_data['tps']
1043+ , iteration_data['rwreqps']
1044+ , iteration_data['deadlocksps']
1045+ , iteration_data['min_req_lat_ms']
1046+ , iteration_data['max_req_lat_ms']
1047+ , iteration_data['avg_req_lat_ms']
1048+ , iteration_data['95p_req_lat_ms']
1049+ )
1050+ retcode, result= execute_query(query, dsn_string=dsn_string)
1051+ return result
1052+
1053+def getSysbenchRegressionReport(run_id, test_data, dsn_string, diff_check_data = None):
1054+ """Returns a textual report of the regression over a series of runs"""
1055+
1056+ # TODO: Allow for comparing one branch name vs. another...
1057+ # add greater flexibility for working with such data
1058+ bzr_branch = 'staging'
1059+ report_notation = ''
1060+ full_commentary = None
1061+
1062+ (last_5_revs, last_20_revs)= get5and20RevisionRanges(run_id, bzr_branch, test_data, dsn_string)
1063+ report_text= """====================================================================================================
1064+SYSBENCH BENCHMARK REPORT %s
1065+====================================================================================================
1066+MACHINE: %s
1067+RUN ID: %d
1068+RUN DATE: %s
1069+WORKLOAD: %s
1070+SERVER: %s
1071+VERSION: %s
1072+REVISION: %d
1073+COMMENT: %s
1074+====================================================================================================
1075+
1076+TRENDING OVER LAST 5 runs %s
1077+""" % (
1078+ report_notation
1079+ , test_data['test_machine']
1080+ , run_id
1081+ , test_data['run_date']
1082+ , test_data['config_name']
1083+ , test_data['test_server_type']
1084+ , bzr_branch
1085+ , int(test_data['test_server_revno'])
1086+ , test_data['test_server_comment']
1087+ , report_notation
1088+)
1089+
1090+ report_text= report_text + "%-6s %-7s %-17s %-10s %-10s %-10s %-10s %-10s" % ("Conc","TPS","% Diff from Avg","Diff","Min","Max","Avg","STD")
1091+ report_text= report_text + """
1092+====================================================================================================
1093+"""
1094+ if len(last_5_revs) > 0:
1095+ results= getRegressionData(run_id, last_5_revs, dsn_string)
1096+ for result in results:
1097+ report_text= report_text + "%-6s %6s %12s %10s %10s %10s %10s %10s\n" % tuple(result)
1098+ report_text= report_text + """====================================================================================================
1099+
1100+TRENDING OVER Last 20 runs %s
1101+
1102+""" % (
1103+ report_notation
1104+)
1105+
1106+ report_text= report_text + "%-6s %-7s %-17s %-10s %-10s %-10s %-10s %-10s" % ("Conc","TPS","% Diff from Avg","Diff","Min","Max","Avg","STD")
1107+ report_text= report_text + """
1108+====================================================================================================
1109+"""
1110+ if len(last_20_revs) > 0:
1111+ results= getRegressionData(run_id, last_20_revs, dsn_string)
1112+ for result in results:
1113+ report_text= report_text + "%-6s %6s %12s %10s %10s %10s %10s %10s\n" % tuple(result)
1114+
1115+ report_text= report_text + """====================================================================================================
1116+
1117+TRENDING OVER ALL runs %s
1118+
1119+""" % (
1120+ report_notation
1121+)
1122+ report_text= report_text + "%-6s %-7s %-17s %-10s %-10s %-10s %-10s %-10s" % ("Conc","TPS","% Diff from Avg","Diff","Min","Max","Avg","STD")
1123+ report_text= report_text + """
1124+====================================================================================================
1125+"""
1126+
1127+ results= getAllRegressionData(test_data['test_machine'], bzr_branch, run_id, dsn_string, test_data)
1128+ for result in results:
1129+ report_text= report_text + "%-6s %6s %12s %10s %10s %10s %10s %10s\n" % tuple(result)
1130+ report_text= report_text + "===================================================================================================="
1131+
1132+ if full_commentary:
1133+ report_text= report_text + """
1134+FULL REVISION COMMENTARY:
1135+
1136+%s""" % full_commentary
1137+
1138+ if diff_check_data:
1139+ report_text += "ERROR: The following tests were flagged as performance regressions.\n"
1140+ for datum in diff_check_data:
1141+ report_text += "%s\n" %datum
1142+ return report_text
1143+
1144+def get5and20RevisionRanges(run_id, bzr_branch, test_data, dsn_string):
1145+ """ Return a tuple with 2 ranges of run_id values for the last 5 and 20 runs
1146+ Ported from drizzle-automation
1147+ TODO: Further refactor / eliminate this
1148+
1149+ """
1150+
1151+ query = """ SELECT
1152+ run_id
1153+ FROM bench_config c
1154+ NATURAL JOIN bench_runs r
1155+ WHERE c.name = '%s'
1156+ AND r.server = '%s'
1157+ AND r.version LIKE '%s%%'
1158+ AND r.run_id <= %d
1159+ ORDER BY run_id DESC
1160+ LIMIT 20
1161+ """ % ( test_data['config_name']
1162+ , test_data['test_machine']
1163+ , bzr_branch
1164+ , run_id
1165+ )
1166+ retcode, results = execute_query(query, dsn_string=dsn_string)
1167+ results_data = []
1168+ for result in results:
1169+ cur_run_id= int(result[0])
1170+ results_data.append(str(cur_run_id))
1171+ last_5_revs = results_data[0:5]
1172+ last_20_revs = results_data[0:20]
1173+ return (last_5_revs, last_20_revs)
1174+
1175+def getRegressionData(run_id, id_range, dsn_string):
1176+ query= """ SELECT
1177+ i.concurrency
1178+ , ROUND(AVG(i.tps), 2) AS tps
1179+ , IF (AVG(i.tps) >= agg.avg_tps
1180+ , CONCAT('+', ROUND(((AVG(i.tps) - agg.avg_tps) / agg.avg_tps) * 100, 2), '%%')
1181+ , CONCAT('-', ROUND(((agg.avg_tps - AVG(i.tps)) / agg.avg_tps) * 100, 2), '%%')
1182+ ) as pct_diff_from_avg
1183+ , ROUND((AVG(i.tps) - agg.avg_tps), 2) as diff_from_avg
1184+ , ROUND(agg.min_tps, 2) AS min_tps
1185+ , ROUND(agg.max_tps, 2) AS max_tps
1186+ , ROUND(agg.avg_tps, 2) AS avg_tps
1187+ , FORMAT(ROUND(agg.stddev_tps, 2),2) AS stddev_tps
1188+ FROM bench_config c
1189+ NATURAL JOIN bench_runs r
1190+ NATURAL JOIN sysbench_run_iterations i
1191+ INNER JOIN (
1192+ SELECT
1193+ concurrency
1194+ , MIN(tps) as min_tps
1195+ , MAX(tps) as max_tps
1196+ , AVG(tps) as avg_tps
1197+ , STDDEV(tps) as stddev_tps
1198+ FROM sysbench_run_iterations iter
1199+ WHERE run_id IN (%s)
1200+ GROUP BY concurrency
1201+ ) AS agg
1202+ ON i.concurrency = agg.concurrency
1203+ WHERE r.run_id = %d
1204+ GROUP BY i.concurrency
1205+ ORDER BY i.concurrency
1206+ """ %( ",".join(id_range)
1207+ , run_id)
1208+ retcode, result= execute_query(query, dsn_string=dsn_string)
1209+ return result
1210+
1211+def getAllRegressionData(server_name, bzr_branch, run_id, dsn_string, test_data):
1212+ query = """ SELECT
1213+ i.concurrency
1214+ , ROUND(AVG(i.tps), 2) AS tps
1215+ , IF (AVG(i.tps) >= agg.avg_tps
1216+ , CONCAT('+', ROUND(((AVG(i.tps) - agg.avg_tps) / agg.avg_tps) * 100, 2), '%%')
1217+ , CONCAT('-', ROUND(((agg.avg_tps - AVG(i.tps)) / agg.avg_tps) * 100, 2), '%%')
1218+ ) as pct_diff_from_avg
1219+ , ROUND((AVG(i.tps) - agg.avg_tps), 2) as diff_from_avg
1220+ , ROUND(agg.min_tps, 2) AS min_tps
1221+ , ROUND(agg.max_tps, 2) AS max_tps
1222+ , ROUND(agg.avg_tps, 2) AS avg_tps
1223+ , FORMAT(ROUND(agg.stddev_tps, 2),2) AS stddev_tps
1224+ FROM bench_config c
1225+ NATURAL JOIN bench_runs r
1226+ NATURAL JOIN sysbench_run_iterations i
1227+ INNER JOIN (
1228+ SELECT
1229+ iter.concurrency
1230+ , MIN(tps) as min_tps
1231+ , MAX(tps) as max_tps
1232+ , AVG(tps) as avg_tps
1233+ , STDDEV(tps) as stddev_tps
1234+ FROM bench_config conf
1235+ NATURAL JOIN bench_runs runs
1236+ NATURAL JOIN sysbench_run_iterations iter
1237+ WHERE conf.name = '%s'
1238+ AND runs.server = '%s'
1239+ AND runs.version LIKE '%s%%'
1240+ GROUP BY iter.concurrency
1241+ ) AS agg
1242+ ON i.concurrency = agg.concurrency
1243+ WHERE r.run_id = %d
1244+ GROUP BY i.concurrency
1245+ ORDER BY i.concurrency
1246+ """ % ( test_data['config_name']
1247+ , server_name
1248+ , bzr_branch
1249+ , run_id
1250+ )
1251+
1252+ retcode, result= execute_query(query, dsn_string=dsn_string)
1253+ return result
1254+
1255+def getConfigId(dsn_string, test_data):
1256+ """Returns the integer ID of the configuration name used in this run."""
1257+
1258+ # If we have not already done so, we query the local DB for the ID
1259+ # matching this sqlbench config name. If none is there, we insert
1260+ # a new record in the bench_config table and return the newly generated
1261+ # identifier.
1262+ benchmark_name = test_data['config_name']
1263+ query = "SELECT config_id FROM bench_config WHERE name = '%s'" %benchmark_name
1264+ retcode, result= execute_query(query, dsn_string=dsn_string)
1265+ if len(result) == 0:
1266+ # Insert a new record for this config and return the new ID...
1267+ query = "INSERT INTO bench_config (config_id, name) VALUES (NULL, '%s')" %benchmark_name
1268+ retcode, result= execute_query(query, dsn_string=dsn_string)
1269+ return getConfigId(dsn_string, test_data)
1270+ else:
1271+ config_id= int(result[0][0])
1272+ return config_id
1273+
1274+def getNextRunId(dsn_string):
1275+ """Returns a new run identifier from the database.
1276+ The run ID is used in logging the results of the run iterations.
1277+
1278+ """
1279+
1280+ query = "SELECT MAX(run_id) as new_run_id FROM bench_runs"
1281+ retcode, result= execute_query(query, dsn_string=dsn_string)
1282+ if result[0][0] >= 1:
1283+ new_run_id= int(result[0][0]) + 1
1284+ else:
1285+ new_run_id= 1
1286+ return new_run_id
1287+
1288+
1289+def getSysbenchReport(run,fetch):
1290+ """returns the report of the last sysbench test executed"""
1291+
1292+ report="""==============================
1293+ SYSBENCH REGRESSION REPORT
1294+==============================
1295+CONCURRENCY : %d
1296+ITERATIONS : %d
1297+MODE : %s
1298+TPS : %f %f
1299+MIN_REQ_LAT_MS : %f %f
1300+MAX_REQ_LAT_MS : %f %f
1301+AVG_REQ_LAT_MS : %f %f
1302+95P_REQ_LAT_MS : %f %f
1303+RWREQPS : %f %f
1304+DEADLOCKSPS : %f %f
1305+=============================
1306+ """ % (fetch['concurrency'],
1307+ fetch['iteration']+1,
1308+ iteration_data['mode'],
1309+ iteration_data['tps'], iteration_data['tps']-fetch['tps'],
1310+ iteration_data['min_req_lat_ms'],iteration_data['min_req_lat_ms']-fetch['min_req_lat_ms'],
1311+ iteration_data['max_req_lat_ms'],iteration_data['max_req_lat_ms']-fetch['max_req_lat_ms'],
1312+ iteration_data['avg_req_lat_ms'],iteration_data['avg_req_lat_ms']-fetch['avg_req_lat_ms'],
1313+ iteration_data['95p_req_lat_ms'],iteration_data['95p_req_lat_ms']-fetch['95p_req_lat_ms'],
1314+ iteration_data['rwreqps'], iteration_data['rwreqps']-fetch['rwreqps'],
1315+ iteration_data['deadlocksps'], iteration_data['deadlocksps']-fetch['deadlocksps']
1316+ )
1317+ return report
1318
1319
1320=== modified file 'tests/qp_tests/crashme/crashme_test.py'
1321--- tests/qp_tests/crashme/crashme_test.py 2012-04-03 19:14:33 +0000
1322+++ tests/qp_tests/crashme/crashme_test.py 2012-08-21 17:59:20 +0000
1323@@ -19,23 +19,45 @@
1324 # along with this program; if not, write to the Free Software
1325 # Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
1326
1327-import unittest
1328-import subprocess
1329+import os
1330+import time
1331
1332 from lib.util.crashme_methods import execute_crashme
1333+from lib.util.mysqlBaseTestCase import mysqlBaseTestCase
1334+from lib.util.mailing_report import kewpieSendMail
1335+from lib.opts.test_run_options import parse_qp_options
1336
1337 server_requirements = [[]]
1338 servers = []
1339 server_manager = None
1340 test_executor = None
1341
1342-class basicTest(unittest.TestCase):
1343+class basicTest(mysqlBaseTestCase):
1344
1345 def test_runCrashme(self):
1346- test_cmd = "$SQLBENCH_DIR/crash-me --server=drizzled --host=127.0.0.1 --force --dir=$DRIZZLE_TEST_WORKDIR --connect-options=port=$MASTER_MYPORT --verbose --debug --user=root --batch-mode"
1347- test_status, retcode, output = execute_crashme(test_cmd, test_executor, servers)
1348+ master_server = servers[0]
1349+ system_manager = test_executor.system_manager
1350+ test_cmd = [ "%s/crash-me " %(os.path.join(system_manager.testdir, 'test_tools/sql-bench'))
1351+ , "--server=drizzled "
1352+ , "--host=127.0.0.1 "
1353+ , "--force "
1354+ , "--dir=%s " %system_manager.workdir
1355+ , "--connect-options=port=%s " %(master_server.master_port)
1356+ , "--verbose "
1357+ , "--debug "
1358+ , "--user=root "
1359+ , "--batch-mode"
1360+ ]
1361+ test_cmd = " ".join(test_cmd)
1362+ test_status, retcode, output = execute_crashme(test_cmd, test_executor, master_server)
1363 self.assertEqual(retcode, 0, msg = output)
1364 self.assertEqual(test_status, 'pass', msg = output)
1365+ print "output:%s" % test_status
1366+ crashme_report="BENCHMARK EXECUTED: crashme\nTEST RESULT:%s"%test_status
1367+ print crashme_report
1368+
1369+ if mail_tgt:
1370+ kewpieSendMail(test_executor,mail_tgt,test_status)
1371
1372 def tearDown(self):
1373 server_manager.reset_servers(test_executor.name)
1374
1375=== added directory 'tests/qp_tests/drizzleslap'
1376=== added file 'tests/qp_tests/drizzleslap/drizzleslap_test.py'
1377--- tests/qp_tests/drizzleslap/drizzleslap_test.py 1970-01-01 00:00:00 +0000
1378+++ tests/qp_tests/drizzleslap/drizzleslap_test.py 2012-08-21 17:59:20 +0000
1379@@ -0,0 +1,195 @@
1380+#! /usr/bin/env python
1381+# -*- mode: python; indent-tabs-mode: nil; -*-
1382+# vim:expandtab:shiftwidth=2:tabstop=2:smarttab:
1383+#
1384+# Copyright (C) 2011 Patrick Crews
1385+# Copyright (C) 2012 Sharan Kumar
1386+#
1387+#
1388+# This program is free software; you can redistribute it and/or modify
1389+# it under the terms of the GNU General Public License as published by
1390+# the Free Software Foundation; either version 2 of the License, or
1391+# (at your option) any later version.
1392+#
1393+# This program is distributed in the hope that it will be useful,
1394+# but WITHOUT ANY WARRANTY; without even the implied warranty of
1395+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
1396+# GNU General Public License for more details.
1397+#
1398+# You should have received a copy of the GNU General Public License
1399+# along with this program; if not, write to the Free Software
1400+# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
1401+
1402+import unittest
1403+import subprocess
1404+import time
1405+import re
1406+
1407+from lib.util.drizzleslap_methods import prepare_drizzleslap
1408+from lib.util.drizzleslap_methods import execute_drizzleslap
1409+from lib.util.drizzleslap_methods import process_drizzleslap_output
1410+from lib.util.mysqlBaseTestCase import mysqlBaseTestCase
1411+from lib.util.database_connect import results_db_connect
1412+from lib.util.mailing_report import kewpieSendMail
1413+from lib.opts.test_run_options import parse_qp_options
1414+
1415+# TODO: make server_options vary depending on the type of server being used here
1416+# drizzle options
1417+#server_requirements = [['innodb.buffer-pool-size=256M innodb.log-file-size=64M innodb.log-buffer-size=8M innodb.thread-concurrency=0 innodb.additional-mem-pool-size=16M table-open-cache=4096 table-definition-cache=4096 mysql-protocol.max-connections=2048']]
1418+# mysql options
1419+#server_requirements = [['innodb_buffer_pool_size=256M innodb_log_file_size=64M innodb_log_buffer_size=8M innodb_thread_concurrency=0 innodb_additional_mem_pool_size=16M table_open_cache=4096 table_definition_cache=4096 max_connections=2048']]
1420+server_requirements = [[]]
1421+servers = []
1422+server_manager = None
1423+test_executor = None
1424+
1425+class basicTest(mysqlBaseTestCase):
1426+
1427+ def test_drizzleslap(self):
1428+ self.logging = test_executor.logging
1429+ master_server = servers[0]
1430+
1431+ # test group
1432+ test_groups = ['guid','guid-scale',
1433+ 'key','key-scale',
1434+ 'mixed','mixed-commit','mixed-commit-scale','mixed-scale',
1435+ 'scan','scan-scale',
1436+ 'update','update-commit','update-commit-scale','update-scale',
1437+ 'write','write-commit','write-commit-scale','write-scale']
1438+
1439+ # test options specific to each test group
1440+ test_options = {'guid':" --auto-generate-sql-guid-primary --auto-generate-sql-load-type=write --number-of-queries=100000",
1441+ 'guid-scale':" --auto-generate-sql-guid-primary --auto-generate-sql-load-type=write --auto-generate-sql-execute-number=1000",
1442+ 'key':"--auto-generate-sql-add-autoincrement --auto-generate-sql-load-type=key --number-of-queries=100000",
1443+ 'key-scale':"--auto-generate-sql-add-autoincrement --auto-generate-sql-load-type=key --auto-generate-sql-execute-number=1000",
1444+ 'mixed':" --auto-generate-sql-add-autoincrement --auto-generate-sql-load-type=mixed --number-of-queries=100000",
1445+ 'mixed-commit':" --auto-generate-sql-add-autoincrement --auto-generate-sql-load-type=mixed --number-of-queries=100000 --commit=8",
1446+ 'mixed-commit-scale':" --auto-generate-sql-add-autoincrement --auto-generate-sql-load-type=mixed --auto-generate-sql-execute-number=1000 --commit=8",
1447+ 'mixed-scale':" --auto-generate-sql-add-autoincrement --auto-generate-sql-load-type=mixed --auto-generate-sql-execute-number=1000",
1448+ 'scan':" --auto-generate-sql-add-autoincrement --auto-generate-sql-load-type=read --number-of-queries=100000",
1449+ 'scan-scale':" --auto-generate-sql-add-autoincrement --auto-generate-sql-load-type=read --auto-generate-sql-execute-number=1000",
1450+ 'update':" --auto-generate-sql-add-autoincrement --auto-generate-sql-load-type=update --number-of-queries=100000 --auto-generate-sql-write-number=50000",
1451+ 'update-commit':" --auto-generate-sql-add-autoincrement --auto-generate-sql-load-type=update --number-of-queries=100000 --auto-generate-sql-write-number=50000 --commit=8 ",
1452+ 'update-commit-scale':" --auto-generate-sql-add-autoincrement --auto-generate-sql-load-type=update --auto-generate-sql-execute-number=1000 --auto-generate-sql-write-number=50000 --commit=8 ",
1453+ 'update-scale':" --auto-generate-sql-add-autoincrement --auto-generate-sql-load-type=update --auto-generate-sql-execute-number=1000 --auto-generate-sql-write-number=50000",
1454+ 'write':"--auto-generate-sql-add-autoincrement --auto-generate-sql-load-type=write --number-of-queries=100000",
1455+ 'write-commit':" --auto-generate-sql-add-autoincrement --auto-generate-sql-load-type=write --number-of-queries=100000 --commit=8",
1456+ 'write-commit-scale':"--auto-generate-sql-add-autoincrement --auto-generate-sql-load-type=write --auto-generate-sql-execute-number=1000 --commit=8 ",
1457+ 'write-scale':" --auto-generate-sql-add-autoincrement --auto-generate-sql-load-type=write --auto-generate-sql-execute-number=1000"
1458+ }
1459+
1460+ # our base test command
1461+ test_cmd = [ "drizzleslap_tests"
1462+ , "--%s-table-engine=innodb" %master_server.type
1463+ , "--%s-user=root" %master_server.type
1464+ , "--%s-db=test" %master_server.type
1465+ , "--%s-port=%d" %(master_server.type, master_server.master_port)
1466+ , "--%s-host=localhost" %master_server.type
1467+ , "--db-driver=%s" %master_server.type
1468+ ]
1469+
1470+ if master_server.type == 'drizzle':
1471+ test_cmd.append("--drizzle-mysql=on")
1472+ if master_server.type == 'mysql':
1473+ test_cmd.append("--mysql-socket=%s" %master_server.socket_file)
1474+
1475+ # We sleep for a minute to wait
1476+ time.sleep(10)
1477+ # how many times to run drizzleslap at each concurrency
1478+ iterations = 10
1479+
1480+ # setting concurreny for drizzleslap. This concurrency is fixed
1481+ concurrencies = [50]
1482+
1483+
1484+ # we setup once. This is a readwrite test and we don't
1485+ # alter the test bed once it is created
1486+ exec_cmd = " ".join(test_cmd)
1487+ retcode, output = prepare_drizzleslap(test_executor, exec_cmd)
1488+ err_msg = ("drizzleslap 'prepare' phase failed.\n"
1489+ "retcode: %d"
1490+ "output: %s" %(retcode,output))
1491+ self.assertEqual(retcode, 0, msg = err_msg)
1492+
1493+ # start the test!
1494+ for concurrency in concurrencies:
1495+
1496+ for group in test_groups:
1497+ exec_cmd = " ".join(test_cmd)
1498+ exec_cmd = exec_cmd.join(test_options[group])
1499+ exec_cmd += "--num-threads=%d" %concurrency
1500+
1501+ for test_iteration in range(iterations):
1502+
1503+ retcode, output = execute_drizzleslap(test_executor, exec_cmd)
1504+ self.assertEqual(retcode, 0, msg = output)
1505+ parsed_output = process_drizzleslap_output(output)
1506+ self.logging.info(parsed_output)
1507+
1508+ #gathering the data from the output
1509+# TODO
1510+ regexes={
1511+ 'run_id':re.compile()
1512+ , 'engine_name':re.compile()
1513+ , 'test_name':re.compile()
1514+ , 'queries_average':re.compile()
1515+ , 'queries_min':re.compile()
1516+ , 'queries_max':re.compile()
1517+ , 'total_time':re.compile()
1518+ , 'stddev':re.compile()
1519+ , 'iterations':re.compile()
1520+ , 'concurrency':re.compile()
1521+ , 'concurrency2':re.compile()
1522+ , 'queries_per_client':re.compile()
1523+ }
1524+
1525+ run={}
1526+ for line in output.split("\n"):
1527+ for key in regexes:
1528+ result=regexes[key].match(line)
1529+ if result:
1530+ run[key]=float(result.group(1))
1531+
1532+
1533+ # fetching test results from results_db database
1534+ sql_select="SELECT * FROM drizzleslap_run_iterations WHERE concurrency=%d AND iteration=%d" % (concurrency,test_iteration)
1535+ self.logging.info("dsn_string:%s" % dsn_string)
1536+ fetch=results_db_connect(dsn_string,"select",sql_select)
1537+ fetch['concurrency']=concurrency
1538+ fetch['iteration']=test_iteration
1539+
1540+ # deleting record with current concurrency and iteration
1541+ if fetch['concurrency']==concurrency and fetch['iteration']==test_iteration:
1542+ sql_delete="DELETE FROM drizzleslap_run_iterations WHERE concurrency=%d AND iteration=%d" % (concurrency,test_iteration)
1543+ results_db_connect(dsn_string,"delete",sql_delete)
1544+
1545+ # updating the results_db database with test results
1546+ # it for historical comparison over the life of the code...
1547+ sql_insert="""INSERT INTO drizzleslap_run_iterations VALUES (%d,'%s','%s',%0.3f,%0.3f,%0.3f,%0.3f,%0.3f,%d,%d,%d,%d )""" % (
1548+ run['run_id'],
1549+ run['engine_name'],
1550+ run['test_name'],
1551+ float(run['queries_avg']),
1552+ float(run['queries_min']),
1553+ float(run['queries_max']),
1554+ float(run['total_time']),
1555+ float(run['stddev']),
1556+ int(run['iterations']),
1557+ int(run['concurrency']),
1558+ int(run['concurrency2']),
1559+ int(run['queries_per_client']) )
1560+
1561+ results_db_connect(dsn_string,"insert",sql_insert)
1562+
1563+#TODO get drizzleslap test result ( should modify this and add util method too )
1564+ #getting test result as report for sysbench
1565+ sys_report=getSysbenchReport(run,fetch)
1566+
1567+ #mailing sysbench report
1568+ if mail_tgt:
1569+ kewpieSendMail(test_executor,mail_tgt,sys_report)
1570+
1571+
1572+ def tearDown(self):
1573+ server_manager.reset_servers(test_executor.name)
1574+
1575
1576=== modified file 'tests/qp_tests/sqlbench/sqlbench_test.py'
1577--- tests/qp_tests/sqlbench/sqlbench_test.py 2011-12-06 21:34:39 +0000
1578+++ tests/qp_tests/sqlbench/sqlbench_test.py 2012-08-21 17:59:20 +0000
1579@@ -24,6 +24,8 @@
1580
1581 from lib.util.sqlbench_methods import execute_sqlbench
1582 from lib.util.mysqlBaseTestCase import mysqlBaseTestCase
1583+from lib.util.mailing_report import kewpieSendMail
1584+from lib.opts.test_run_options import parse_qp_options
1585
1586 server_requirements = [[]]
1587 servers = []
1588@@ -39,6 +41,10 @@
1589 self.assertEqual(retcode, 0, msg = output)
1590 self.assertEqual(test_status, 'pass', msg = output)
1591
1592+ # sending test report via mail
1593+ if mail_tgt:
1594+ kewpieSendMail(test_executor,mail_tgt,test_status)
1595+
1596 def tearDown(self):
1597 server_manager.reset_servers(test_executor.name)
1598
1599
1600=== modified file 'tests/qp_tests/sysbench/sysbench_readonly_test.py'
1601--- tests/qp_tests/sysbench/sysbench_readonly_test.py 2012-06-19 20:58:24 +0000
1602+++ tests/qp_tests/sysbench/sysbench_readonly_test.py 2012-08-21 17:59:20 +0000
1603@@ -3,6 +3,7 @@
1604 # vim:expandtab:shiftwidth=2:tabstop=2:smarttab:
1605 #
1606 # Copyright (C) 2011 Patrick Crews
1607+# Copyright (C) 2012 M.Sharan Kumar
1608 #
1609 #
1610 # This program is free software; you can redistribute it and/or modify
1611@@ -19,31 +20,40 @@
1612 # along with this program; if not, write to the Free Software
1613 # Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
1614
1615-import unittest
1616+import re
1617+import time
1618+import socket
1619 import subprocess
1620-import time
1621+import datetime
1622+from copy import deepcopy
1623
1624 from lib.util.sysbench_methods import prepare_sysbench
1625 from lib.util.sysbench_methods import execute_sysbench
1626 from lib.util.sysbench_methods import process_sysbench_output
1627-from lib.util.mysqlBaseTestCase import mysqlBaseTestCase
1628+from lib.util.sysbench_methods import sysbench_db_analysis
1629+from lib.util.sysbench_methods import getSysbenchReport
1630+from lib.util.sysbenchTestCase import sysbenchTestCase
1631+from lib.util.database_connect import results_db_connect
1632+from lib.util.mailing_report import sendMail
1633
1634 # TODO: make server_options vary depending on the type of server being used here
1635 # drizzle options
1636-#server_requirements = [['innodb.buffer-pool-size=256M innodb.log-file-size=64M innodb.log-buffer-size=8M innodb.thread-concurrency=0 innodb.additional-mem-pool-size=16M table-open-cache=4096 table-definition-cache=4096 mysql-protocol.max-connections=2048']]
1637+server_requirements = [['innodb.buffer-pool-size=256M innodb.log-file-size=64M innodb.log-buffer-size=8M innodb.thread-concurrency=0 innodb.additional-mem-pool-size=16M table-open-cache=4096 table-definition-cache=4096 mysql-protocol.max-connections=2048']]
1638+
1639 # mysql options
1640 #server_requirements = [['innodb_buffer_pool_size=256M innodb_log_file_size=64M innodb_log_buffer_size=8M innodb_thread_concurrency=0 innodb_additional_mem_pool_size=16M table_open_cache=4096 table_definition_cache=4096 max_connections=2048']]
1641-server_requirements = [[]]
1642+
1643 servers = []
1644 server_manager = None
1645 test_executor = None
1646
1647-class basicTest(mysqlBaseTestCase):
1648-
1649+class basicTest(sysbenchTestCase):
1650+
1651 def test_sysbench_readonly(self):
1652- self.logging = test_executor.logging
1653+
1654+ # defining the test command
1655 master_server = servers[0]
1656- # our base test command
1657+ self.config_name = 'innodb_1000K_readonly'
1658 test_cmd = [ "sysbench"
1659 , "--max-time=240"
1660 , "--max-requests=0"
1661@@ -58,46 +68,21 @@
1662 , "--%s-host=localhost" %master_server.type
1663 , "--db-driver=%s" %master_server.type
1664 ]
1665-
1666 if master_server.type == 'drizzle':
1667 test_cmd.append("--drizzle-mysql=on")
1668 if master_server.type == 'mysql':
1669 test_cmd.append("--mysql-socket=%s" %master_server.socket_file)
1670-
1671- # We sleep for a minute to wait
1672- time.sleep(10)
1673- # how many times to run sysbench at each concurrency
1674- iterations = 1
1675-
1676- # various concurrencies to use with sysbench
1677- #concurrencies = [16, 32, 64, 128, 256, 512, 1024]
1678- concurrencies = [1, 4, 8 ]
1679+
1680+ # preparing sysbench_readonly test
1681+ self.prepareSysbench(test_cmd,test_executor,servers)
1682
1683 # start the test!
1684- for concurrency in concurrencies:
1685- self.logging.info("Resetting test server...")
1686- for query in ["DROP SCHEMA IF EXISTS test"
1687- ,"CREATE SCHEMA test"
1688- ]:
1689- retcode, result = self.execute_query(query, master_server, schema="INFORMATION_SCHEMA")
1690- test_cmd.append("--num-threads=%d" %concurrency)
1691- # we setup once per concurrency, copying drizzle-automation
1692- # this should likely change and if not for readonly, then definitely
1693- # for readwrite
1694-
1695- exec_cmd = " ".join(test_cmd)
1696- retcode, output = prepare_sysbench(test_executor, exec_cmd)
1697- err_msg = ("sysbench 'prepare' phase failed.\n"
1698- "retcode: %d"
1699- "output: %s" %(retcode,output))
1700- self.assertEqual(retcode, 0, msg = err_msg)
1701-
1702- for test_iteration in range(iterations):
1703- retcode, output = execute_sysbench(test_executor, exec_cmd)
1704- self.assertEqual(retcode, 0, msg = output)
1705- parsed_output = process_sysbench_output(output)
1706- self.logging.info(parsed_output)
1707+ # this method takes care of *running* the test and *saving* the test results
1708+ self.executeSysbench()
1709+
1710+ # reporting the test result
1711+ # this method handles *dsn_string* and *mail_tgt*
1712+ self.reportTestData(dsn_string,mail_tgt)
1713
1714 def tearDown(self):
1715 server_manager.reset_servers(test_executor.name)
1716-
1717
1718=== added file 'tests/qp_tests/sysbench/sysbench_readwrite_test.py'
1719--- tests/qp_tests/sysbench/sysbench_readwrite_test.py 1970-01-01 00:00:00 +0000
1720+++ tests/qp_tests/sysbench/sysbench_readwrite_test.py 2012-08-21 17:59:20 +0000
1721@@ -0,0 +1,89 @@
1722+#! /usr/bin/env python
1723+# -*- mode: python; indent-tabs-mode: nil; -*-
1724+# vim:expandtab:shiftwidth=2:tabstop=2:smarttab:
1725+#
1726+# Copyright (C) 2011 Patrick Crews
1727+# Copyright (C) 2012 M.Sharan Kumar
1728+#
1729+#
1730+# This program is free software; you can redistribute it and/or modify
1731+# it under the terms of the GNU General Public License as published by
1732+# the Free Software Foundation; either version 2 of the License, or
1733+# (at your option) any later version.
1734+#
1735+# This program is distributed in the hope that it will be useful,
1736+# but WITHOUT ANY WARRANTY; without even the implied warranty of
1737+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
1738+# GNU General Public License for more details.
1739+#
1740+# You should have received a copy of the GNU General Public License
1741+# along with this program; if not, write to the Free Software
1742+# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
1743+
1744+import re
1745+import time
1746+import socket
1747+import subprocess
1748+import datetime
1749+from copy import deepcopy
1750+
1751+from lib.util.sysbench_methods import prepare_sysbench
1752+from lib.util.sysbench_methods import execute_sysbench
1753+from lib.util.sysbench_methods import process_sysbench_output
1754+from lib.util.sysbench_methods import sysbench_db_analysis
1755+from lib.util.sysbench_methods import getSysbenchReport
1756+from lib.util.sysbenchTestCase import sysbenchTestCase
1757+from lib.util.database_connect import results_db_connect
1758+from lib.util.mailing_report import sendMail
1759+
1760+# TODO: make server_options vary depending on the type of server being used here
1761+# drizzle options
1762+server_requirements = [['innodb.buffer-pool-size=256M innodb.log-file-size=64M innodb.log-buffer-size=8M innodb.thread-concurrency=0 innodb.additional-mem-pool-size=16M table-open-cache=4096 table-definition-cache=4096 mysql-protocol.max-connections=2048']]
1763+
1764+# mysql options
1765+#server_requirements = [['innodb_buffer_pool_size=256M innodb_log_file_size=64M innodb_log_buffer_size=8M innodb_thread_concurrency=0 innodb_additional_mem_pool_size=16M table_open_cache=4096 table_definition_cache=4096 max_connections=2048']]
1766+
1767+servers = []
1768+server_manager = None
1769+test_executor = None
1770+
1771+class basicTest(sysbenchTestCase):
1772+
1773+ def test_sysbench_readonly(self):
1774+
1775+ # defining the test command
1776+ master_server = servers[0]
1777+ self.config_name = 'innodb_1000K_readonly'
1778+ test_cmd = [ "sysbench"
1779+ , "--max-time=240"
1780+ , "--max-requests=0"
1781+ , "--test=oltp"
1782+ , "--db-ps-mode=disable"
1783+ , "--%s-table-engine=innodb" %master_server.type
1784+ , "--oltp-read-only=off"
1785+ , "--oltp-table-size=1000000"
1786+ , "--%s-user=root" %master_server.type
1787+ , "--%s-db=test" %master_server.type
1788+ , "--%s-port=%d" %(master_server.type, master_server.master_port)
1789+ , "--%s-host=localhost" %master_server.type
1790+ , "--db-driver=%s" %master_server.type
1791+ ]
1792+
1793+ if master_server.type == 'drizzle':
1794+ test_cmd.append("--drizzle-mysql=on")
1795+ if master_server.type == 'mysql':
1796+ test_cmd.append("--mysql-socket=%s" %master_server.socket_file)
1797+
1798+ # preparing sysbench_readonly test
1799+ self.prepareSysbench(test_cmd,test_executor,servers)
1800+
1801+ # start the test!
1802+ # this method takes care of *running* the test and *saving* the test results
1803+ self.executeSysbench()
1804+
1805+ # reporting the test result
1806+ # this method handles *dsn_string* and *mail_tgt*
1807+ self.reportTestData(dsn_string,mail_tgt)
1808+
1809+ def tearDown(self):
1810+ server_manager.reset_servers(test_executor.name)
1811
1812=== added file 'tests/std_data/sysbench_db.sql'
1813--- tests/std_data/sysbench_db.sql 1970-01-01 00:00:00 +0000
1814+++ tests/std_data/sysbench_db.sql 2012-08-21 17:59:20 +0000
1815@@ -0,0 +1,107 @@
1816+-- MySQL dump 10.13 Distrib 5.1.63, for debian-linux-gnu (x86_64)
1817+--
1818+-- Host: localhost Database: drizzle_stats
1819+-- ------------------------------------------------------
1820+-- Server version 5.1.63-0ubuntu0.10.04.1
1821+
1822+/*!40101 SET @OLD_CHARACTER_SET_CLIENT=@@CHARACTER_SET_CLIENT */;
1823+/*!40101 SET @OLD_CHARACTER_SET_RESULTS=@@CHARACTER_SET_RESULTS */;
1824+/*!40101 SET @OLD_COLLATION_CONNECTION=@@COLLATION_CONNECTION */;
1825+/*!40101 SET NAMES utf8 */;
1826+/*!40103 SET @OLD_TIME_ZONE=@@TIME_ZONE */;
1827+/*!40103 SET TIME_ZONE='+00:00' */;
1828+/*!40014 SET @OLD_UNIQUE_CHECKS=@@UNIQUE_CHECKS, UNIQUE_CHECKS=0 */;
1829+/*!40014 SET @OLD_FOREIGN_KEY_CHECKS=@@FOREIGN_KEY_CHECKS, FOREIGN_KEY_CHECKS=0 */;
1830+/*!40101 SET @OLD_SQL_MODE=@@SQL_MODE, SQL_MODE='NO_AUTO_VALUE_ON_ZERO' */;
1831+/*!40111 SET @OLD_SQL_NOTES=@@SQL_NOTES, SQL_NOTES=0 */;
1832+
1833+--
1834+-- Table structure for table `bench_config`
1835+--
1836+
1837+DROP TABLE IF EXISTS `bench_config`;
1838+/*!40101 SET @saved_cs_client = @@character_set_client */;
1839+/*!40101 SET character_set_client = utf8 */;
1840+CREATE TABLE `bench_config` (
1841+ `config_id` int(11) NOT NULL AUTO_INCREMENT,
1842+ `name` varchar(255) NOT NULL,
1843+ PRIMARY KEY (`config_id`)
1844+) ENGINE=InnoDB AUTO_INCREMENT=2 DEFAULT CHARSET=latin1;
1845+/*!40101 SET character_set_client = @saved_cs_client */;
1846+
1847+--
1848+-- Dumping data for table `bench_config`
1849+--
1850+
1851+LOCK TABLES `bench_config` WRITE;
1852+/*!40000 ALTER TABLE `bench_config` DISABLE KEYS */;
1853+/*!40000 ALTER TABLE `bench_config` ENABLE KEYS */;
1854+UNLOCK TABLES;
1855+
1856+--
1857+-- Table structure for table `bench_runs`
1858+--
1859+
1860+DROP TABLE IF EXISTS `bench_runs`;
1861+/*!40101 SET @saved_cs_client = @@character_set_client */;
1862+/*!40101 SET character_set_client = utf8 */;
1863+CREATE TABLE `bench_runs` (
1864+ `run_id` int(11) NOT NULL AUTO_INCREMENT,
1865+ `config_id` int(11) NOT NULL,
1866+ `server` varchar(20) NOT NULL,
1867+ `version` varchar(60) DEFAULT NULL,
1868+ `run_date` datetime NOT NULL,
1869+ PRIMARY KEY (`run_id`)
1870+) ENGINE=InnoDB AUTO_INCREMENT=2 DEFAULT CHARSET=latin1;
1871+/*!40101 SET character_set_client = @saved_cs_client */;
1872+
1873+--
1874+-- Dumping data for table `bench_runs`
1875+--
1876+
1877+LOCK TABLES `bench_runs` WRITE;
1878+/*!40000 ALTER TABLE `bench_runs` DISABLE KEYS */;
1879+/*!40000 ALTER TABLE `bench_runs` ENABLE KEYS */;
1880+UNLOCK TABLES;
1881+
1882+--
1883+-- Table structure for table `sysbench_run_iterations`
1884+--
1885+
1886+DROP TABLE IF EXISTS `sysbench_run_iterations`;
1887+/*!40101 SET @saved_cs_client = @@character_set_client */;
1888+/*!40101 SET character_set_client = utf8 */;
1889+CREATE TABLE `sysbench_run_iterations` (
1890+ `run_id` int(11) NOT NULL,
1891+ `concurrency` int(11) NOT NULL,
1892+ `iteration` int(11) NOT NULL,
1893+ `tps` decimal(13,2) NOT NULL,
1894+ `read_write_req_per_second` decimal(13,2) NOT NULL,
1895+ `deadlocks_per_second` decimal(5,2) NOT NULL,
1896+ `min_req_latency_ms` decimal(10,2) NOT NULL,
1897+ `avg_req_latency_ms` decimal(10,2) NOT NULL,
1898+ `max_req_latency_ms` decimal(10,2) NOT NULL,
1899+ `95p_req_latency_ms` decimal(10,2) NOT NULL,
1900+ PRIMARY KEY (`run_id`,`concurrency`,`iteration`)
1901+) ENGINE=InnoDB DEFAULT CHARSET=latin1;
1902+/*!40101 SET character_set_client = @saved_cs_client */;
1903+
1904+--
1905+-- Dumping data for table `sysbench_run_iterations`
1906+--
1907+
1908+LOCK TABLES `sysbench_run_iterations` WRITE;
1909+/*!40000 ALTER TABLE `sysbench_run_iterations` DISABLE KEYS */;
1910+/*!40000 ALTER TABLE `sysbench_run_iterations` ENABLE KEYS */;
1911+UNLOCK TABLES;
1912+/*!40103 SET TIME_ZONE=@OLD_TIME_ZONE */;
1913+
1914+/*!40101 SET SQL_MODE=@OLD_SQL_MODE */;
1915+/*!40014 SET FOREIGN_KEY_CHECKS=@OLD_FOREIGN_KEY_CHECKS */;
1916+/*!40014 SET UNIQUE_CHECKS=@OLD_UNIQUE_CHECKS */;
1917+/*!40101 SET CHARACTER_SET_CLIENT=@OLD_CHARACTER_SET_CLIENT */;
1918+/*!40101 SET CHARACTER_SET_RESULTS=@OLD_CHARACTER_SET_RESULTS */;
1919+/*!40101 SET COLLATION_CONNECTION=@OLD_COLLATION_CONNECTION */;
1920+/*!40111 SET SQL_NOTES=@OLD_SQL_NOTES */;
1921+
1922+-- Dump completed on 2012-08-14 12:50:51

Subscribers

People subscribed via source and target branches

to all changes: