Launchpad itself

Merge lp:~jml/launchpad/different-ec2-mail into lp:launchpad

  1. different-ec2-mail
  2. Merge into devel
Proposed by Jonathan Lange
Status: Merged
Merged at revision: 11390
Proposed branch: lp:~jml/launchpad/different-ec2-mail
Merge into: lp:launchpad
Diff against target: 1807 lines (+1222/-361)
4 files modified
lib/devscripts/ec2test/remote.py (+567/-323)
lib/devscripts/ec2test/testrunner.py (+11/-6)
lib/devscripts/ec2test/tests/remote_daemonization_test.py (+49/-0)
lib/devscripts/ec2test/tests/test_remote.py (+595/-32)
To merge this branch: bzr merge lp:~jml/launchpad/different-ec2-mail
Reviewer Review Type Date Requested Status
Jelmer Vernooij (community) code Approve
Review via email: mp+32905@code.launchpad.net

Commit message

Refactor ec2test/remote.py and add a whole bunch of tests.

Description of the change

This branch took a really long time and doesn't do what I wanted it to originally.

It busts up ec2test/remote.py, the script that runs on EC2 instances which runs tests, gathers output and sends emails to PQM and developers. It's also responsible for shutting down the instance when running in detached mode.

The refactoring is pretty severe, so you might be better off comparing the old file and the new file side by side, rather than looking at the diff.

I've avoided behavioral changes, even when they would be simple, since mixing them with refactoring generally only causes woe. However, I may have made the occasional slip.

Once I refactored the script and roughly demonstrated that it works by doing a couple of test runs, I added a whole bunch of tests. I wanted to try to reach full test coverage, but I ran out of steam. The untested bits are flagged as such.

To summarize the refactoring:
 * The large chunk of code in an 'if __name__ == "__main__"' block has been
   split into parse_options() and main() functions.

 * TestOnMergeRunner has merged into BaseTestRunner and then split up into
   three classes.
   1. EC2Runner, which knows how to run arbitrary stuff and do
      daemonization & shutdown around it.
   2. LaunchpadTester, which knows how to run Launchpad tests and gather
      results.
   3. Request, which knows all about the test run we've been asked to do,
      and how to send information back to the end user.

 * WebTestLogger has been beefed up so that its clients don't have to
   fiddle directly with the files it manages. It is now also responsible
   for deciding what to do with the test result once it gets it.

 * SummaryResult has been tweaked to be way more testable.

 * A general renaming of internal attributes and methods to have
   underscore prefixes.

I'm not wedded to any of the names of the classes, but I do think the breakdown of responsibilities makes sense.

In addition to the refactoring, I added a --not-really option to "ec2 test", because it was very helpful for me when debugging. Since the ec2 commands are already laboring under a heavy burden of options and flags that are rarely (if ever) used, I'd be happy to drop it.

Let me know if you have any questions, and thanks for reviewing such a large branch.

To post a comment you must log in.
Revision history for this message
Jelmer Vernooij (jelmer) wrote :

Nice to see the class structure rearranged and great to see much more test coverage.

As mentioned on IRC, two minor points:

<jelmer> jml: is there any reason you use Branch.open_containing vs Branch.open ?
<jml> jelmer, I don't, it's preserved from the original. I reckon we should use Branch.open & would happily change the code.
<jelmer> jml: --not-really doesn't really explain to me what it's not doing. What about --no-tests or --test-setup-only ?
<jml> jelmer, I'm going to delete the whole thing. it's buggy anyway

With that in mind, r=me.

review: Approve (code)

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== modified file 'lib/devscripts/ec2test/remote.py'
--- lib/devscripts/ec2test/remote.py 2010-06-08 03:01:14 +0000
+++ lib/devscripts/ec2test/remote.py 2010-08-18 18:07:59 +0000
@@ -1,9 +1,21 @@
1#!/usr/bin/env python1#!/usr/bin/env python
2# Run tests in a daemon.
3#
4# Copyright 2009 Canonical Ltd. This software is licensed under the2# Copyright 2009 Canonical Ltd. This software is licensed under the
5# GNU Affero General Public License version 3 (see the file LICENSE).3# GNU Affero General Public License version 3 (see the file LICENSE).
64
5"""Run tests in a daemon.
6
7 * `EC2Runner` handles the daemonization and instance shutdown.
8
9 * `Request` knows everything about the test request we're handling (e.g.
10 "test merging foo-bar-bug-12345 into db-devel").
11
12 * `LaunchpadTester` knows how to actually run the tests and gather the
13 results. It uses `SummaryResult` and `FlagFallStream` to do so.
14
15 * `WebTestLogger` knows how to display the results to the user, and is given
16 the responsibility of handling the results that `LaunchpadTester` gathers.
17"""
18
7from __future__ import with_statement19from __future__ import with_statement
820
9__metatype__ = type21__metatype__ = type
@@ -11,6 +23,7 @@
11import datetime23import datetime
12from email import MIMEMultipart, MIMEText24from email import MIMEMultipart, MIMEText
13from email.mime.application import MIMEApplication25from email.mime.application import MIMEApplication
26import errno
14import gzip27import gzip
15import optparse28import optparse
16import os29import os
@@ -38,29 +51,38 @@
38class SummaryResult(unittest.TestResult):51class SummaryResult(unittest.TestResult):
39 """Test result object used to generate the summary."""52 """Test result object used to generate the summary."""
4053
41 double_line = '=' * 70 + '\n'54 double_line = '=' * 70
42 single_line = '-' * 70 + '\n'55 single_line = '-' * 70
4356
44 def __init__(self, output_stream):57 def __init__(self, output_stream):
45 super(SummaryResult, self).__init__()58 super(SummaryResult, self).__init__()
46 self.stream = output_stream59 self.stream = output_stream
4760
48 def printError(self, flavor, test, error):61 def _formatError(self, flavor, test, error):
49 """Print an error to the output stream."""62 return '\n'.join(
50 self.stream.write(self.double_line)63 [self.double_line,
51 self.stream.write('%s: %s\n' % (flavor, test))64 '%s: %s' % (flavor, test),
52 self.stream.write(self.single_line)65 self.single_line,
53 self.stream.write('%s\n' % (error,))66 error,
54 self.stream.flush()67 ''])
5568
56 def addError(self, test, error):69 def addError(self, test, error):
57 super(SummaryResult, self).addError(test, error)70 super(SummaryResult, self).addError(test, error)
58 self.printError('ERROR', test, self._exc_info_to_string(error, test))71 self.stream.write(
72 self._formatError(
73 'ERROR', test, self._exc_info_to_string(error, test)))
5974
60 def addFailure(self, test, error):75 def addFailure(self, test, error):
61 super(SummaryResult, self).addFailure(test, error)76 super(SummaryResult, self).addFailure(test, error)
62 self.printError(77 self.stream.write(
63 'FAILURE', test, self._exc_info_to_string(error, test))78 self._formatError(
79 'FAILURE', test, self._exc_info_to_string(error, test)))
80
81 def stopTest(self, test):
82 super(SummaryResult, self).stopTest(test)
83 # At the very least, we should be sure that a test's output has been
84 # completely displayed once it has stopped.
85 self.stream.flush()
6486
6587
66class FlagFallStream:88class FlagFallStream:
@@ -93,52 +115,118 @@
93 self._stream.flush()115 self._stream.flush()
94116
95117
96class BaseTestRunner:118class EC2Runner:
97119 """Runs generic code in an EC2 instance.
98 def __init__(self, email=None, pqm_message=None, public_branch=None,120
99 public_branch_revno=None, test_options=None):121 Handles daemonization, instance shutdown, and email in the case of
100 self.email = email122 catastrophic failure.
101 self.pqm_message = pqm_message123 """
102 self.public_branch = public_branch124
103 self.public_branch_revno = public_branch_revno125 # XXX: JonathanLange 2010-08-17: EC2Runner needs tests.
104 self.test_options = test_options126
105127 # The number of seconds we give this script to clean itself up, and for
106 # Configure paths.128 # 'ec2 test --postmortem' to grab control if needed. If we don't give
107 self.lp_dir = os.path.join(os.path.sep, 'var', 'launchpad')129 # --postmortem enough time to log in via SSH and take control, then this
108 self.tmp_dir = os.path.join(self.lp_dir, 'tmp')130 # server will begin to shutdown on its own.
109 self.test_dir = os.path.join(self.lp_dir, 'test')131 #
110 self.sourcecode_dir = os.path.join(self.test_dir, 'sourcecode')132 # (FWIW, "grab control" means sending SIGTERM to this script's process id,
111133 # thus preventing fail-safe shutdown.)
112 # Set up logging.134 SHUTDOWN_DELAY = 60
113 self.logger = WebTestLogger(135
114 self.test_dir,136 def __init__(self, daemonize, pid_filename, shutdown_when_done,
115 self.public_branch,137 emails=None):
116 self.public_branch_revno,138 """Make an EC2Runner.
117 self.sourcecode_dir139
118 )140 :param daemonize: Whether or not we will daemonize.
119141 :param pid_filename: The filename to store the pid in.
120 # Daemonization options.142 :param shutdown_when_done: Whether or not to shut down when the tests
121 self.pid_filename = os.path.join(self.lp_dir, 'ec2test-remote.pid')143 are done.
122 self.daemonized = False144 :param emails: The email address(es) to send catastrophic failure
123145 messages to. If not provided, the error disappears into the ether.
124 def daemonize(self):146 """
147 self._should_daemonize = daemonize
148 self._pid_filename = pid_filename
149 self._shutdown_when_done = shutdown_when_done
150 self._emails = emails
151 self._daemonized = False
152
153 def _daemonize(self):
125 """Turn the testrunner into a forked daemon process."""154 """Turn the testrunner into a forked daemon process."""
126 # This also writes our pidfile to disk to a specific location. The155 # This also writes our pidfile to disk to a specific location. The
127 # ec2test.py --postmortem command will look there to find our PID,156 # ec2test.py --postmortem command will look there to find our PID,
128 # in order to control this process.157 # in order to control this process.
129 daemonize(self.pid_filename)158 daemonize(self._pid_filename)
130 self.daemonized = True159 self._daemonized = True
131160
132 def remove_pidfile(self):161 def _shutdown_instance(self):
133 if os.path.exists(self.pid_filename):162 """Shut down this EC2 instance."""
134 os.remove(self.pid_filename)163 # Make sure our process is daemonized, and has therefore disconnected
135164 # the controlling terminal. This also disconnects the ec2test.py SSH
136 def ignore_line(self, line):165 # connection, thus signalling ec2test.py that it may now try to take
137 """Return True if the line should be excluded from the summary log.166 # control of the server.
138167 if not self._daemonized:
139 Defaults to False.168 # We only want to do this if we haven't already been daemonized.
169 # Nesting daemons is bad.
170 self._daemonize()
171
172 time.sleep(self.SHUTDOWN_DELAY)
173
174 # We'll only get here if --postmortem didn't kill us. This is our
175 # fail-safe shutdown, in case the user got disconnected or suffered
176 # some other mishap that would prevent them from shutting down this
177 # server on their own.
178 subprocess.call(['sudo', 'shutdown', '-P', 'now'])
179
180 def run(self, name, function, *args, **kwargs):
181 try:
182 if self._should_daemonize:
183 print 'Starting %s daemon...' % (name,)
184 self._daemonize()
185
186 return function(*args, **kwargs)
187 except:
188 config = bzrlib.config.GlobalConfig()
189 # Handle exceptions thrown by the test() or daemonize() methods.
190 if self._emails:
191 bzrlib.email_message.EmailMessage.send(
192 config, config.username(),
193 self._emails, '%s FAILED' % (name,),
194 traceback.format_exc())
195 raise
196 finally:
197 # When everything is over, if we've been ask to shut down, then
198 # make sure we're daemonized, then shutdown. Otherwise, if we're
199 # daemonized, just clean up the pidfile.
200 if self._shutdown_when_done:
201 self._shutdown_instance()
202 elif self._daemonized:
203 # It would be nice to clean up after ourselves, since we won't
204 # be shutting down.
205 remove_pidfile(self._pid_filename)
206 else:
207 # We're not a daemon, and we're not shutting down. The user
208 # most likely started this script manually, from a shell
209 # running on the instance itself.
210 pass
211
212
213class LaunchpadTester:
214 """Runs Launchpad tests and gathers their results in a useful way."""
215
216 # XXX: JonathanLange 2010-08-17: LaunchpadTester needs tests.
217
218 def __init__(self, logger, test_directory, test_options=()):
219 """Construct a TestOnMergeRunner.
220
221 :param logger: The WebTestLogger to log to.
222 :param test_directory: The directory to run the tests in. We expect
223 this directory to have a fully-functional checkout of Launchpad
224 and its dependent branches.
225 :param test_options: A sequence of options to pass to the test runner.
140 """226 """
141 return False227 self._logger = logger
228 self._test_directory = test_directory
229 self._test_options = ' '.join(test_options)
142230
143 def build_test_command(self):231 def build_test_command(self):
144 """Return the command that will execute the test suite.232 """Return the command that will execute the test suite.
@@ -148,7 +236,8 @@
148236
149 Subclasses must provide their own implementation of this method.237 Subclasses must provide their own implementation of this method.
150 """238 """
151 raise NotImplementedError239 command = ['make', 'check', 'TESTOPTS="%s"' % self._test_options]
240 return command
152241
153 def test(self):242 def test(self):
154 """Run the tests, log the results.243 """Run the tests, log the results.
@@ -157,170 +246,327 @@
157 have completed. If necessary, submits the branch to PQM, and mails246 have completed. If necessary, submits the branch to PQM, and mails
158 the user the test results.247 the user the test results.
159 """248 """
160 # We need to open the log files here because earlier calls to249 self._logger.prepare()
161 # os.fork() may have tried to close them.
162 self.logger.prepare()
163
164 out_file = self.logger.out_file
165 summary_file = self.logger.summary_file
166 config = bzrlib.config.GlobalConfig()
167250
168 call = self.build_test_command()251 call = self.build_test_command()
169252
170 try:253 try:
254 self._logger.write_line("Running %s" % (call,))
255 # XXX: JonathanLange 2010-08-18: bufsize=-1 implies "use the
256 # system buffering", when we actually want no buffering at all.
171 popen = subprocess.Popen(257 popen = subprocess.Popen(
172 call, bufsize=-1,258 call, bufsize=-1,
173 stdout=subprocess.PIPE, stderr=subprocess.STDOUT,259 stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
174 cwd=self.test_dir)260 cwd=self._test_directory)
175261
176 self._gather_test_output(popen.stdout, summary_file, out_file)262 self._gather_test_output(popen.stdout, self._logger)
177263
178 # Grab the testrunner exit status.264 exit_status = popen.wait()
179 result = popen.wait()
180
181 if self.pqm_message is not None:
182 subject = self.pqm_message.get('Subject')
183 if result:
184 # failure
185 summary_file.write(
186 '\n\n**NOT** submitted to PQM:\n%s\n' % (subject,))
187 else:
188 # success
189 conn = bzrlib.smtp_connection.SMTPConnection(config)
190 conn.send_email(self.pqm_message)
191 summary_file.write(
192 '\n\nSUBMITTED TO PQM:\n%s\n' % (subject,))
193
194 summary_file.write(
195 '\n(See the attached file for the complete log)\n')
196 except:265 except:
197 summary_file.write('\n\nERROR IN TESTRUNNER\n\n')266 self._logger.error_in_testrunner(sys.exc_info())
198 traceback.print_exc(file=summary_file)267 exit_status = 1
199 result = 1
200 raise268 raise
201 finally:269 finally:
202 # It probably isn't safe to close the log files ourselves,270 self._logger.got_result(not exit_status)
203 # since someone else might try to write to them later.271
204 try:272 def _gather_test_output(self, input_stream, logger):
205 if self.email is not None:273 """Write the testrunner output to the logs."""
206 self.send_email(274 summary_stream = logger.get_summary_stream()
207 result, self.logger.summary_filename,275 result = SummaryResult(summary_stream)
208 self.logger.out_filename, config)276 subunit_server = subunit.TestProtocolServer(result, summary_stream)
209 finally:277 for line in input_stream:
210 summary_file.close()278 subunit_server.lineReceived(line)
211 # we do this at the end because this is a trigger to279 logger.got_line(line)
212 # ec2test.py back at home that it is OK to kill the process280
213 # and take control itself, if it wants to.281
214 self.logger.close_logs()282class Request:
215283 """A request to have a branch tested and maybe landed."""
216 def send_email(self, result, summary_filename, out_filename, config):284
217 """Send an email summarizing the test results.285 def __init__(self, branch_url, revno, local_branch_path, sourcecode_path,
218286 emails=None, pqm_message=None):
219 :param result: True for pass, False for failure.287 """Construct a `Request`.
220 :param summary_filename: The path to the file where the summary288
221 information lives. This will be the body of the email.289 :param branch_url: The public URL to the Launchpad branch we are
222 :param out_filename: The path to the file where the full output290 testing.
223 lives. This will be zipped and attached.291 :param revno: The revision number of the branch we are testing.
224 :param config: A Bazaar configuration object with SMTP details.292 :param local_branch_path: A local path to the Launchpad branch we are
293 testing. This must be a branch of Launchpad with a working tree.
294 :param sourcecode_path: A local path to the sourcecode dependencies
295 directory (normally '$local_branch_path/sourcecode'). This must
296 contain up-to-date copies of all of Launchpad's sourcecode
297 dependencies.
298 :param emails: A list of emails to send the results to. If not
299 provided, no emails are sent.
300 :param pqm_message: The message to submit to PQM. If not provided, we
301 don't submit to PQM.
302 """
303 self._branch_url = branch_url
304 self._revno = revno
305 self._local_branch_path = local_branch_path
306 self._sourcecode_path = sourcecode_path
307 self._emails = emails
308 self._pqm_message = pqm_message
309 # Used for figuring out how to send emails.
310 self._bzr_config = bzrlib.config.GlobalConfig()
311
312 def _send_email(self, message):
313 """Actually send 'message'."""
314 conn = bzrlib.smtp_connection.SMTPConnection(self._bzr_config)
315 conn.send_email(message)
316
317 def get_trunk_details(self):
318 """Return (branch_url, revno) for trunk."""
319 branch = bzrlib.branch.Branch.open(self._local_branch_path)
320 return branch.get_parent().encode('utf-8'), branch.revno()
321
322 def get_branch_details(self):
323 """Return (branch_url, revno) for the branch we're merging in.
324
325 If we're not merging in a branch, but instead just testing a trunk,
326 then return None.
327 """
328 tree = bzrlib.workingtree.WorkingTree.open(self._local_branch_path)
329 parent_ids = tree.get_parent_ids()
330 if len(parent_ids) < 2:
331 return None
332 return self._branch_url.encode('utf-8'), self._revno
333
334 def _last_segment(self, url):
335 """Return the last segment of a URL."""
336 return url.strip('/').split('/')[-1]
337
338 def get_nick(self):
339 """Get the nick of the branch we are testing."""
340 details = self.get_branch_details()
341 if not details:
342 details = self.get_trunk_details()
343 url, revno = details
344 return self._last_segment(url)
345
346 def get_merge_description(self):
347 """Get a description of the merge request.
348
349 If we're merging a branch, return '$SOURCE_NICK => $TARGET_NICK', if
350 we're just running tests for a trunk branch without merging return
351 '$TRUNK_NICK'.
352 """
353 # XXX: JonathanLange 2010-08-17: Not actually used yet. I think it
354 # would be a great thing to have in the subject of the emails we
355 # receive.
356 source = self.get_branch_details()
357 if not source:
358 return self.get_nick()
359 target = self.get_trunk_details()
360 return '%s => %s' % (
361 self._last_segment(source[0]), self._last_segment(target[0]))
362
363 def get_summary_commit(self):
364 """Get a message summarizing the change from the commit log.
365
366 Returns the last commit message of the merged branch, or None.
367 """
368 # XXX: JonathanLange 2010-08-17: I don't actually know why we are
369 # using this commit message as a summary message. It's used in the
370 # test logs and the EC2 hosted web page.
371 branch = bzrlib.branch.Branch.open(self._local_branch_path)
372 tree = bzrlib.workingtree.WorkingTree.open(self._local_branch_path)
373 parent_ids = tree.get_parent_ids()
374 if len(parent_ids) == 1:
375 return None
376 summary = (
377 branch.repository.get_revision(parent_ids[1]).get_summary())
378 return summary.encode('utf-8')
379
380 def _build_report_email(self, successful, body_text, full_log_gz):
381 """Build a MIME email summarizing the test results.
382
383 :param successful: True for pass, False for failure.
384 :param body_text: The body of the email to send to the requesters.
385 :param full_log_gz: A gzip of the full log.
225 """386 """
226 message = MIMEMultipart.MIMEMultipart()387 message = MIMEMultipart.MIMEMultipart()
227 message['To'] = ', '.join(self.email)388 message['To'] = ', '.join(self._emails)
228 message['From'] = config.username()389 message['From'] = self._bzr_config.username()
229 subject = 'Test results: %s' % (result and 'FAILURE' or 'SUCCESS')390 if successful:
391 status = 'SUCCESS'
392 else:
393 status = 'FAILURE'
394 subject = 'Test results: %s' % (status,)
230 message['Subject'] = subject395 message['Subject'] = subject
231396
232 # Make the body.397 # Make the body.
233 with open(summary_filename, 'r') as summary_fd:398 body = MIMEText.MIMEText(body_text, 'plain', 'utf8')
234 summary = summary_fd.read()
235 body = MIMEText.MIMEText(summary, 'plain', 'utf8')
236 body['Content-Disposition'] = 'inline'399 body['Content-Disposition'] = 'inline'
237 message.attach(body)400 message.attach(body)
238401
239 # gzip up the full log.
240 fd, path = tempfile.mkstemp()
241 os.close(fd)
242 gz = gzip.open(path, 'wb')
243 full_log = open(out_filename, 'rb')
244 gz.writelines(full_log)
245 gz.close()
246
247 # Attach the gzipped log.402 # Attach the gzipped log.
248 zipped_log = MIMEApplication(open(path, 'rb').read(), 'x-gzip')403 zipped_log = MIMEApplication(full_log_gz, 'x-gzip')
249 zipped_log.add_header(404 zipped_log.add_header(
250 'Content-Disposition', 'attachment',405 'Content-Disposition', 'attachment',
251 filename='%s.log.gz' % self.get_nick())406 filename='%s.log.gz' % self.get_nick())
252 message.attach(zipped_log)407 message.attach(zipped_log)
253408 return message
254 bzrlib.smtp_connection.SMTPConnection(config).send_email(message)409
255410 def send_report_email(self, successful, body_text, full_log_gz):
256 def get_nick(self):411 """Send an email summarizing the test results.
257 """Return the nick name of the branch that we are testing."""412
258 return self.public_branch.strip('/').split('/')[-1]413 :param successful: True for pass, False for failure.
259414 :param body_text: The body of the email to send to the requesters.
260 def _gather_test_output(self, input_stream, summary_file, out_file):415 :param full_log_gz: A gzip of the full log.
261 """Write the testrunner output to the logs."""416 """
262 # Only write to stdout if we are running as the foreground process.417 message = self._build_report_email(successful, body_text, full_log_gz)
263 echo_to_stdout = not self.daemonized418 self._send_email(message)
264 result = SummaryResult(summary_file)419
265 subunit_server = subunit.TestProtocolServer(result, summary_file)420 def iter_dependency_branches(self):
266 for line in input_stream:421 """Iterate through the Bazaar branches we depend on."""
267 subunit_server.lineReceived(line)422 for name in sorted(os.listdir(self._sourcecode_path)):
268 out_file.write(line)423 path = os.path.join(self._sourcecode_path, name)
269 out_file.flush()424 if os.path.isdir(path):
270 if echo_to_stdout:425 try:
271 sys.stdout.write(line)426 branch = bzrlib.branch.Branch.open(path)
272 sys.stdout.flush()427 except bzrlib.errors.NotBranchError:
273428 continue
274429 yield name, branch.get_parent(), branch.revno()
275class TestOnMergeRunner(BaseTestRunner):430
276 """Executes the Launchpad test_on_merge.py test suite."""431 def submit_to_pqm(self, successful):
277432 """Submit this request to PQM, if successful & configured to do so."""
278 def build_test_command(self):433 if not self._pqm_message:
279 """See BaseTestRunner.build_test_command()."""434 return
280 command = ['make', 'check', 'TESTOPTS=' + self.test_options]435 subject = self._pqm_message.get('Subject')
281 return command436 if successful:
437 self._send_email(self._pqm_message)
438 return subject
439
440 @property
441 def wants_email(self):
442 """Do the requesters want emails sent to them?"""
443 return bool(self._emails)
282444
283445
284class WebTestLogger:446class WebTestLogger:
285 """Logs test output to disk and a simple web page."""447 """Logs test output to disk and a simple web page."""
286448
287 def __init__(self, test_dir, public_branch, public_branch_revno,449 def __init__(self, full_log_filename, summary_filename, index_filename,
288 sourcecode_dir):450 request, echo_to_stdout):
289 """ Class initialiser """451 """Construct a WebTestLogger.
290 self.test_dir = test_dir452
291 self.public_branch = public_branch453 Because this writes an HTML file with links to the summary and full
292 self.public_branch_revno = public_branch_revno454 logs, you should construct this object with
293 self.sourcecode_dir = sourcecode_dir455 `WebTestLogger.make_in_directory`, which guarantees that the files
294456 are available in the correct locations.
295 self.www_dir = os.path.join(os.path.sep, 'var', 'www')457
296 self.out_filename = os.path.join(self.www_dir, 'current_test.log')458 :param full_log_filename: Path to a file that will have the full
297 self.summary_filename = os.path.join(self.www_dir, 'summary.log')459 log output written to it. The file will be overwritten.
298 self.index_filename = os.path.join(self.www_dir, 'index.html')460 :param summary_file: Path to a file that will have a human-readable
299461 summary written to it. The file will be overwritten.
300 # We will set up the preliminary bits of the web-accessible log462 :param index_file: Path to a file that will have an HTML page
301 # files. "out" holds all stdout and stderr; "summary" holds filtered463 written to it. The file will be overwritten.
302 # output; and "index" holds an index page.464 :param request: A `Request` object representing the thing that's being
303 self.out_file = None465 tested.
304 self.summary_file = None466 :param echo_to_stdout: Whether or not we should echo output to stdout.
305 self.index_file = None467 """
306468 self._full_log_filename = full_log_filename
307 def open_logs(self):469 self._summary_filename = summary_filename
308 """Open all of our log files for writing."""470 self._index_filename = index_filename
309 self.out_file = open(self.out_filename, 'w')471 self._request = request
310 self.summary_file = open(self.summary_filename, 'w')472 self._echo_to_stdout = echo_to_stdout
311 self.index_file = open(self.index_filename, 'w')473
312474 @classmethod
313 def flush_logs(self):475 def make_in_directory(cls, www_dir, request, echo_to_stdout):
314 """Flush all of our log file buffers."""476 """Make a logger that logs to specific files in `www_dir`.
315 self.out_file.flush()477
316 self.summary_file.flush()478 :param www_dir: The directory in which to log the files:
317 self.index_file.flush()479 current_test.log, summary.log and index.html. These files
318480 will be overwritten.
319 def close_logs(self):481 :param request: A `Request` object representing the thing that's being
320 """Closes all of the open log file handles."""482 tested.
321 self.out_file.close()483 :param echo_to_stdout: Whether or not we should echo output to stdout.
322 self.summary_file.close()484 """
323 self.index_file.close()485 files = [
486 os.path.join(www_dir, 'current_test.log'),
487 os.path.join(www_dir, 'summary.log'),
488 os.path.join(www_dir, 'index.html')]
489 files.extend([request, echo_to_stdout])
490 return cls(*files)
491
492 def error_in_testrunner(self, exc_info):
493 """Called when there is a catastrophic error in the test runner."""
494 exc_type, exc_value, exc_tb = exc_info
495 # XXX: JonathanLange 2010-08-17: This should probably log to the full
496 # log as well.
497 summary = self.get_summary_stream()
498 summary.write('\n\nERROR IN TESTRUNNER\n\n')
499 traceback.print_exception(exc_type, exc_value, exc_tb, file=summary)
500 summary.flush()
501
502 def get_index_contents(self):
503 """Return the contents of the index.html page."""
504 return self._get_contents(self._index_filename)
505
506 def get_full_log_contents(self):
507 """Return the contents of the complete log."""
508 return self._get_contents(self._full_log_filename)
509
510 def get_summary_contents(self):
511 """Return the contents of the summary log."""
512 return self._get_contents(self._summary_filename)
513
514 def get_summary_stream(self):
515 """Return a stream that, when written to, writes to the summary."""
516 return open(self._summary_filename, 'a')
517
518 def got_line(self, line):
519 """Called when we get a line of output from our child processes."""
520 self._write_to_filename(self._full_log_filename, line)
521 if self._echo_to_stdout:
522 sys.stdout.write(line)
523 sys.stdout.flush()
524
525 def _get_contents(self, filename):
526 """Get the full contents of 'filename'."""
527 try:
528 return open(filename, 'r').read()
529 except IOError, e:
530 if e.errno == errno.ENOENT:
531 return ''
532
533 def got_result(self, successful):
534 """The tests are done and the results are known."""
535 self._handle_pqm_submission(successful)
536 if self._request.wants_email:
537 self._write_to_filename(
538 self._summary_filename,
539 '\n(See the attached file for the complete log)\n')
540 summary = self.get_summary_contents()
541 full_log_gz = gzip_data(self.get_full_log_contents())
542 self._request.send_report_email(successful, summary, full_log_gz)
543
544 def _handle_pqm_submission(self, successful):
545 subject = self._request.submit_to_pqm(successful)
546 if not subject:
547 return
548 self.write_line('')
549 self.write_line('')
550 if successful:
551 self.write_line('SUBMITTED TO PQM:')
552 else:
553 self.write_line('**NOT** submitted to PQM:')
554 self.write_line(subject)
555
556 def _write_to_filename(self, filename, msg):
557 fd = open(filename, 'a')
558 fd.write(msg)
559 fd.flush()
560 fd.close()
561
562 def _write(self, msg):
563 """Write to the summary and full log file."""
564 self._write_to_filename(self._full_log_filename, msg)
565 self._write_to_filename(self._summary_filename, msg)
566
567 def write_line(self, msg):
568 """Write to the summary and full log file with a newline."""
569 self._write(msg + '\n')
324570
325 def prepare(self):571 def prepare(self):
326 """Prepares the log files on disk.572 """Prepares the log files on disk.
@@ -328,20 +574,22 @@
328 Writes three log files: the raw output log, the filtered "summary"574 Writes three log files: the raw output log, the filtered "summary"
329 log file, and a HTML index page summarizing the test run paramters.575 log file, and a HTML index page summarizing the test run paramters.
330 """576 """
331 self.open_logs()577 # XXX: JonathanLange 2010-07-18: Mostly untested.
332578 log = self.write_line
333 out_file = self.out_file579
334 summary_file = self.summary_file580 # Clear the existing index file.
335 index_file = self.index_file581 index = open(self._index_filename, 'w')
336582 index.truncate(0)
337 def write(msg):583 index.close()
338 msg += '\n'584
339 summary_file.write(msg)585 def add_to_html(html):
340 out_file.write(msg)586 return self._write_to_filename(
587 self._index_filename, textwrap.dedent(html))
588
341 msg = 'Tests started at approximately %(now)s UTC' % {589 msg = 'Tests started at approximately %(now)s UTC' % {
342 'now': datetime.datetime.utcnow().strftime(590 'now': datetime.datetime.utcnow().strftime(
343 '%a, %d %b %Y %H:%M:%S')}591 '%a, %d %b %Y %H:%M:%S')}
344 index_file.write(textwrap.dedent('''\592 add_to_html('''\
345 <html>593 <html>
346 <head>594 <head>
347 <title>Testing</title>595 <title>Testing</title>
@@ -353,86 +601,76 @@
353 <li><a href="summary.log">Summary results</a></li>601 <li><a href="summary.log">Summary results</a></li>
354 <li><a href="current_test.log">Full results</a></li>602 <li><a href="current_test.log">Full results</a></li>
355 </ul>603 </ul>
356 ''' % (msg,)))604 ''' % (msg,))
357 write(msg)605 log(msg)
358606
359 index_file.write(textwrap.dedent('''\607 add_to_html('''\
360 <h2>Branches Tested</h2>608 <h2>Branches Tested</h2>
361 '''))609 ''')
362610
363 # Describe the trunk branch.611 # Describe the trunk branch.
364 branch = bzrlib.branch.Branch.open_containing(self.test_dir)[0]612 trunk, trunk_revno = self._request.get_trunk_details()
365 msg = '%(trunk)s, revision %(trunk_revno)d\n' % {613 msg = '%s, revision %d\n' % (trunk, trunk_revno)
366 'trunk': branch.get_parent().encode('utf-8'),614 add_to_html('''\
367 'trunk_revno': branch.revno()}
368 index_file.write(textwrap.dedent('''\
369 <p><strong>%s</strong></p>615 <p><strong>%s</strong></p>
370 ''' % (escape(msg),)))616 ''' % (escape(msg),))
371 write(msg)617 log(msg)
372 tree = bzrlib.workingtree.WorkingTree.open(self.test_dir)
373 parent_ids = tree.get_parent_ids()
374618
375 # Describe the merged branch.619 branch_details = self._request.get_branch_details()
376 if len(parent_ids) == 1:620 if not branch_details:
377 index_file.write('<p>(no merged branch)</p>\n')621 add_to_html('<p>(no merged branch)</p>\n')
378 write('(no merged branch)')622 log('(no merged branch)')
379 else:623 else:
380 summary = (624 branch_name, branch_revno = branch_details
381 branch.repository.get_revision(parent_ids[1]).get_summary())625 data = {'name': branch_name,
382 data = {'name': self.public_branch.encode('utf-8'),626 'revno': branch_revno,
383 'revno': self.public_branch_revno,627 'commit': self._request.get_summary_commit()}
384 'commit': summary.encode('utf-8')}
385 msg = ('%(name)s, revision %(revno)d '628 msg = ('%(name)s, revision %(revno)d '
386 '(commit message: %(commit)s)\n' % data)629 '(commit message: %(commit)s)\n' % data)
387 index_file.write(textwrap.dedent('''\630 add_to_html('''\
388 <p>Merged with<br />%(msg)s</p>631 <p>Merged with<br />%(msg)s</p>
389 ''' % {'msg': escape(msg)}))632 ''' % {'msg': escape(msg)})
390 write("Merged with")633 log("Merged with")
391 write(msg)634 log(msg)
392635
393 index_file.write('<dl>\n')636 add_to_html('<dl>\n')
394 write('\nDEPENDENCY BRANCHES USED\n')637 log('\nDEPENDENCY BRANCHES USED\n')
395 for name in os.listdir(self.sourcecode_dir):638 for name, branch, revno in self._request.iter_dependency_branches():
396 path = os.path.join(self.sourcecode_dir, name)639 data = {'name': name, 'branch': branch, 'revno': revno}
397 if os.path.isdir(path):640 log(
398 try:641 '- %(name)s\n %(branch)s\n %(revno)d\n' % data)
399 branch = bzrlib.branch.Branch.open_containing(path)[0]642 escaped_data = {'name': escape(name),
400 except bzrlib.errors.NotBranchError:643 'branch': escape(branch),
401 continue644 'revno': revno}
402 data = {'name': name,645 add_to_html('''\
403 'branch': branch.get_parent(),646 <dt>%(name)s</dt>
404 'revno': branch.revno()}647 <dd>%(branch)s</dd>
405 write(648 <dd>%(revno)s</dd>
406 '- %(name)s\n %(branch)s\n %(revno)d\n' % data)649 ''' % escaped_data)
407 escaped_data = {'name': escape(name),650 add_to_html('''\
408 'branch': escape(branch.get_parent()),
409 'revno': branch.revno()}
410 index_file.write(textwrap.dedent('''\
411 <dt>%(name)s</dt>
412 <dd>%(branch)s</dd>
413 <dd>%(revno)s</dd>
414 ''' % escaped_data))
415 index_file.write(textwrap.dedent('''\
416 </dl>651 </dl>
417 </body>652 </body>
418 </html>'''))653 </html>''')
419 write('\n\nTEST RESULTS FOLLOW\n\n')654 log('\n\nTEST RESULTS FOLLOW\n\n')
420 self.flush_logs()
421655
422656
423def daemonize(pid_filename):657def daemonize(pid_filename):
424 # this seems like the sort of thing that ought to be in the658 # this seems like the sort of thing that ought to be in the
425 # standard library :-/659 # standard library :-/
426 pid = os.fork()660 pid = os.fork()
427 if (pid == 0): # child 1661 if (pid == 0): # Child 1
428 os.setsid()662 os.setsid()
429 pid = os.fork()663 pid = os.fork()
430 if (pid == 0): # child 2664 if (pid == 0): # Child 2, the daemon.
431 pass # lookie, we're ready to do work in the daemon665 pass # lookie, we're ready to do work in the daemon
432 else:666 else:
433 os._exit(0)667 os._exit(0)
434 else:668 else: # Parent
435 # give the pidfile a chance to be written before we exit.669 # Make sure the pidfile is written before we exit, so that people
670 # who've chosen to daemonize can quickly rectify their mistake. Since
671 # the daemon might terminate itself very, very quickly, we cannot poll
672 # for the existence of the pidfile. Instead, we just sleep for a
673 # reasonable amount of time.
436 time.sleep(1)674 time.sleep(1)
437 os._exit(0)675 os._exit(0)
438676
@@ -454,6 +692,38 @@
454 os.dup2(0, 2)692 os.dup2(0, 2)
455693
456694
695def gunzip_data(data):
696 """Decompress 'data'.
697
698 :param data: The gzip data to decompress.
699 :return: The decompressed data.
700 """
701 fd, path = tempfile.mkstemp()
702 os.write(fd, data)
703 os.close(fd)
704 try:
705 return gzip.open(path, 'r').read()
706 finally:
707 os.unlink(path)
708
709
710def gzip_data(data):
711 """Compress 'data'.
712
713 :param data: The data to compress.
714 :return: The gzip-compressed data.
715 """
716 fd, path = tempfile.mkstemp()
717 os.close(fd)
718 gz = gzip.open(path, 'wb')
719 gz.writelines(data)
720 gz.close()
721 try:
722 return open(path).read()
723 finally:
724 os.unlink(path)
725
726
457def write_pidfile(pid_filename):727def write_pidfile(pid_filename):
458 """Write a pidfile for the current process."""728 """Write a pidfile for the current process."""
459 pid_file = open(pid_filename, "w")729 pid_file = open(pid_filename, "w")
@@ -461,7 +731,14 @@
461 pid_file.close()731 pid_file.close()
462732
463733
464if __name__ == '__main__':734def remove_pidfile(pid_filename):
735 if os.path.exists(pid_filename):
736 os.remove(pid_filename)
737
738
739def parse_options(argv):
740 """Make an `optparse.OptionParser` for running the tests remotely.
741 """
465 parser = optparse.OptionParser(742 parser = optparse.OptionParser(
466 usage="%prog [options] [-- test options]",743 usage="%prog [options] [-- test options]",
467 description=("Build and run tests for an instance."))744 description=("Build and run tests for an instance."))
@@ -494,7 +771,11 @@
494 type="int", default=None,771 type="int", default=None,
495 help=('The revision number of the public branch being tested.'))772 help=('The revision number of the public branch being tested.'))
496773
497 options, args = parser.parse_args()774 return parser.parse_args(argv)
775
776
777def main(argv):
778 options, args = parse_options(argv)
498779
499 if options.debug:780 if options.debug:
500 import pdb; pdb.set_trace()781 import pdb; pdb.set_trace()
@@ -504,65 +785,28 @@
504 else:785 else:
505 pqm_message = None786 pqm_message = None
506787
507 runner = TestOnMergeRunner(788 # Locations for Launchpad. These are actually defined by the configuration
508 options.email,789 # of the EC2 image that we use.
509 pqm_message,790 LAUNCHPAD_DIR = '/var/launchpad'
510 options.public_branch,791 TEST_DIR = os.path.join(LAUNCHPAD_DIR, 'test')
511 options.public_branch_revno,792 SOURCECODE_DIR = os.path.join(TEST_DIR, 'sourcecode')
512 ' '.join(args))793
513794 pid_filename = os.path.join(LAUNCHPAD_DIR, 'ec2test-remote.pid')
514 try:795
515 try:796 request = Request(
516 if options.daemon:797 options.public_branch, options.public_branch_revno, TEST_DIR,
517 print 'Starting testrunner daemon...'798 SOURCECODE_DIR, options.email, pqm_message)
518 runner.daemonize()799 # Only write to stdout if we are running as the foreground process.
519800 echo_to_stdout = not options.daemon
520 runner.test()801 logger = WebTestLogger.make_in_directory(
521 except:802 '/var/www', request, echo_to_stdout)
522 config = bzrlib.config.GlobalConfig()803
523 # Handle exceptions thrown by the test() or daemonize() methods.804 runner = EC2Runner(
524 if options.email:805 options.daemon, pid_filename, options.shutdown, options.email)
525 bzrlib.email_message.EmailMessage.send(806
526 config, config.username(),807 tester = LaunchpadTester(logger, TEST_DIR, test_options=args[1:])
527 options.email,808 runner.run("Test runner", tester.test)
528 'Test Runner FAILED', traceback.format_exc())809
529 raise810
530 finally:811if __name__ == '__main__':
531812 main(sys.argv)
532 # When everything is over, if we've been ask to shut down, then
533 # make sure we're daemonized, then shutdown. Otherwise, if we're
534 # daemonized, just clean up the pidfile.
535 if options.shutdown:
536 # Make sure our process is daemonized, and has therefore
537 # disconnected the controlling terminal. This also disconnects
538 # the ec2test.py SSH connection, thus signalling ec2test.py
539 # that it may now try to take control of the server.
540 if not runner.daemonized:
541 # We only want to do this if we haven't already been
542 # daemonized. Nesting daemons is bad.
543 runner.daemonize()
544
545 # Give the script 60 seconds to clean itself up, and 60 seconds
546 # for the ec2test.py --postmortem option to grab control if
547 # needed. If we don't give --postmortem enough time to log
548 # in via SSH and take control, then this server will begin to
549 # shutdown on it's own.
550 #
551 # (FWIW, "grab control" means sending SIGTERM to this script's
552 # process id, thus preventing fail-safe shutdown.)
553 time.sleep(60)
554
555 # We'll only get here if --postmortem didn't kill us. This is
556 # our fail-safe shutdown, in case the user got disconnected
557 # or suffered some other mishap that would prevent them from
558 # shutting down this server on their own.
559 subprocess.call(['sudo', 'shutdown', '-P', 'now'])
560 elif runner.daemonized:
561 # It would be nice to clean up after ourselves, since we won't
562 # be shutting down.
563 runner.remove_pidfile()
564 else:
565 # We're not a daemon, and we're not shutting down. The user most
566 # likely started this script manually, from a shell running on the
567 # instance itself.
568 pass
569813
=== modified file 'lib/devscripts/ec2test/testrunner.py'
--- lib/devscripts/ec2test/testrunner.py 2010-08-13 22:04:58 +0000
+++ lib/devscripts/ec2test/testrunner.py 2010-08-18 18:07:59 +0000
@@ -454,11 +454,8 @@
454 # close ssh connection454 # close ssh connection
455 user_connection.close()455 user_connection.close()
456456
457 def run_tests(self):457 def _build_command(self):
458 self.configure_system()458 """Build the command that we'll use to run the tests."""
459 self.prepare_tests()
460 user_connection = self._instance.connect()
461
462 # Make sure we activate the failsafe --shutdown feature. This will459 # Make sure we activate the failsafe --shutdown feature. This will
463 # make the server shut itself down after the test run completes, or460 # make the server shut itself down after the test run completes, or
464 # if the test harness suffers a critical failure.461 # if the test harness suffers a critical failure.
@@ -496,6 +493,12 @@
496493
497 # Add any additional options for ec2test-remote.py494 # Add any additional options for ec2test-remote.py
498 cmd.extend(['--', self.test_options])495 cmd.extend(['--', self.test_options])
496 return ' '.join(cmd)
497
498 def run_tests(self):
499 self.configure_system()
500 self.prepare_tests()
501
499 self.log(502 self.log(
500 'Running tests... (output is available on '503 'Running tests... (output is available on '
501 'http://%s/)\n' % self._instance.hostname)504 'http://%s/)\n' % self._instance.hostname)
@@ -513,7 +516,9 @@
513516
514 # Run the remote script! Our execution will block here until the517 # Run the remote script! Our execution will block here until the
515 # remote side disconnects from the terminal.518 # remote side disconnects from the terminal.
516 user_connection.perform(' '.join(cmd))519 cmd = self._build_command()
520 user_connection = self._instance.connect()
521 user_connection.perform(cmd)
517 self._running = True522 self._running = True
518523
519 if not self.headless:524 if not self.headless:
520525
=== added file 'lib/devscripts/ec2test/tests/remote_daemonization_test.py'
--- lib/devscripts/ec2test/tests/remote_daemonization_test.py 1970-01-01 00:00:00 +0000
+++ lib/devscripts/ec2test/tests/remote_daemonization_test.py 2010-08-18 18:07:59 +0000
@@ -0,0 +1,49 @@
1# Copyright 2010 Canonical Ltd. This software is licensed under the
2# GNU Affero General Public License version 3 (see the file LICENSE).
3
4"""Script executed by test_remote.py to verify daemonization behaviour.
5
6See TestDaemonizationInteraction.
7"""
8
9import os
10import sys
11import traceback
12
13from devscripts.ec2test.remote import EC2Runner, WebTestLogger
14from devscripts.ec2test.tests.test_remote import TestRequest
15
16PID_FILENAME = os.path.abspath(sys.argv[1])
17DIRECTORY = os.path.abspath(sys.argv[2])
18LOG_FILENAME = os.path.abspath(sys.argv[3])
19
20
21def make_request():
22 """Just make a request."""
23 test = TestRequest('test_wants_email')
24 test.setUp()
25 try:
26 return test.make_request()
27 finally:
28 test.tearDown()
29
30
31def prepare_files(logger):
32 try:
33 logger.prepare()
34 except:
35 # If anything in the above fails, we want to be able to find out about
36 # it. We can't use stdout or stderr because this is a daemon.
37 error_log = open(LOG_FILENAME, 'w')
38 traceback.print_exc(file=error_log)
39 error_log.close()
40
41
42request = make_request()
43os.mkdir(DIRECTORY)
44logger = WebTestLogger.make_in_directory(DIRECTORY, request, True)
45runner = EC2Runner(
46 daemonize=True,
47 pid_filename=PID_FILENAME,
48 shutdown_when_done=False)
49runner.run("test daemonization interaction", prepare_files, logger)
050
=== modified file 'lib/devscripts/ec2test/tests/test_remote.py'
--- lib/devscripts/ec2test/tests/test_remote.py 2010-04-29 09:23:08 +0000
+++ lib/devscripts/ec2test/tests/test_remote.py 2010-08-18 18:07:59 +0000
@@ -5,14 +5,72 @@
55
6__metaclass__ = type6__metaclass__ = type
77
8from email.mime.application import MIMEApplication
9from email.mime.text import MIMEText
10import gzip
11from itertools import izip
12import os
8from StringIO import StringIO13from StringIO import StringIO
14import subprocess
9import sys15import sys
16import tempfile
17import time
18import traceback
10import unittest19import unittest
1120
12from devscripts.ec2test.remote import SummaryResult21from bzrlib.config import GlobalConfig
1322from bzrlib.tests import TestCaseWithTransport
1423
15class TestSummaryResult(unittest.TestCase):24from testtools import TestCase
25from testtools.content import Content
26from testtools.content_type import ContentType
27
28from devscripts.ec2test.remote import (
29 FlagFallStream,
30 gunzip_data,
31 gzip_data,
32 remove_pidfile,
33 Request,
34 SummaryResult,
35 WebTestLogger,
36 write_pidfile,
37 )
38
39
40class TestFlagFallStream(TestCase):
41 """Tests for `FlagFallStream`."""
42
43 def test_doesnt_write_before_flag(self):
44 # A FlagFallStream does not forward any writes before it sees the
45 # 'flag'.
46 stream = StringIO()
47 flag = self.getUniqueString('flag')
48 flagfall = FlagFallStream(stream, flag)
49 flagfall.write('foo')
50 flagfall.flush()
51 self.assertEqual('', stream.getvalue())
52
53 def test_writes_after_flag(self):
54 # After a FlagFallStream sees the flag, it forwards all writes.
55 stream = StringIO()
56 flag = self.getUniqueString('flag')
57 flagfall = FlagFallStream(stream, flag)
58 flagfall.write('foo')
59 flagfall.write(flag)
60 flagfall.write('bar')
61 self.assertEqual('%sbar' % (flag,), stream.getvalue())
62
63 def test_mixed_write(self):
64 # If a single call to write has pre-flagfall and post-flagfall data in
65 # it, then only the post-flagfall data is forwarded to the stream.
66 stream = StringIO()
67 flag = self.getUniqueString('flag')
68 flagfall = FlagFallStream(stream, flag)
69 flagfall.write('foo%sbar' % (flag,))
70 self.assertEqual('%sbar' % (flag,), stream.getvalue())
71
72
73class TestSummaryResult(TestCase):
16 """Tests for `SummaryResult`."""74 """Tests for `SummaryResult`."""
1775
18 def makeException(self, factory=None, *args, **kwargs):76 def makeException(self, factory=None, *args, **kwargs):
@@ -23,50 +81,555 @@
23 except:81 except:
24 return sys.exc_info()82 return sys.exc_info()
2583
26 def test_printError(self):84 def test_formatError(self):
27 # SummaryResult.printError() prints out the name of the test, the kind85 # SummaryResult._formatError() combines the name of the test, the kind
28 # of error and the details of the error in a nicely-formatted way.86 # of error and the details of the error in a nicely-formatted way.
29 stream = StringIO()87 result = SummaryResult(None)
30 result = SummaryResult(stream)88 output = result._formatError('FOO', 'test', 'error')
31 result.printError('FOO', 'test', 'error')89 expected = '%s\nFOO: test\n%s\nerror\n' % (
32 expected = '%sFOO: test\n%serror\n' % (
33 result.double_line, result.single_line)90 result.double_line, result.single_line)
34 self.assertEqual(expected, stream.getvalue())91 self.assertEqual(expected, output)
3592
36 def test_addError(self):93 def test_addError(self):
37 # SummaryResult.addError() prints a nicely-formatted error.94 # SummaryResult.addError doesn't write immediately.
38 #95 stream = StringIO()
39 # First, use printError to build the error text we expect.
40 test = self96 test = self
41 stream = StringIO()97 error = self.makeException()
42 result = SummaryResult(stream)98 result = SummaryResult(stream)
43 error = self.makeException()99 expected = result._formatError(
44 result.printError(
45 'ERROR', test, result._exc_info_to_string(error, test))100 'ERROR', test, result._exc_info_to_string(error, test))
46 expected = stream.getvalue()
47 # Now, call addError and check that it matches.
48 stream = StringIO()
49 result = SummaryResult(stream)
50 result.addError(test, error)101 result.addError(test, error)
51 self.assertEqual(expected, stream.getvalue())102 self.assertEqual(expected, stream.getvalue())
52103
53 def test_addFailure(self):104 def test_addFailure_does_not_write_immediately(self):
54 # SummaryResult.addFailure() prints a nicely-formatted error.105 # SummaryResult.addFailure doesn't write immediately.
55 #106 stream = StringIO()
56 # First, use printError to build the error text we expect.
57 test = self107 test = self
58 stream = StringIO()108 error = self.makeException()
59 result = SummaryResult(stream)109 result = SummaryResult(stream)
60 error = self.makeException(test.failureException)110 expected = result._formatError(
61 result.printError(
62 'FAILURE', test, result._exc_info_to_string(error, test))111 'FAILURE', test, result._exc_info_to_string(error, test))
63 expected = stream.getvalue()
64 # Now, call addFailure and check that it matches.
65 stream = StringIO()
66 result = SummaryResult(stream)
67 result.addFailure(test, error)112 result.addFailure(test, error)
68 self.assertEqual(expected, stream.getvalue())113 self.assertEqual(expected, stream.getvalue())
69114
115 def test_stopTest_flushes_stream(self):
116 # SummaryResult.stopTest() flushes the stream.
117 stream = StringIO()
118 flush_calls = []
119 stream.flush = lambda: flush_calls.append(None)
120 result = SummaryResult(stream)
121 result.stopTest(self)
122 self.assertEqual(1, len(flush_calls))
123
124
125class TestPidfileHelpers(TestCase):
126 """Tests for `write_pidfile` and `remove_pidfile`."""
127
128 def test_write_pidfile(self):
129 fd, path = tempfile.mkstemp()
130 self.addCleanup(os.unlink, path)
131 os.close(fd)
132 write_pidfile(path)
133 self.assertEqual(os.getpid(), int(open(path, 'r').read()))
134
135 def test_remove_pidfile(self):
136 fd, path = tempfile.mkstemp()
137 os.close(fd)
138 write_pidfile(path)
139 remove_pidfile(path)
140 self.assertEqual(False, os.path.exists(path))
141
142 def test_remove_nonexistent_pidfile(self):
143 directory = tempfile.mkdtemp()
144 path = os.path.join(directory, 'doesntexist')
145 remove_pidfile(path)
146 self.assertEqual(False, os.path.exists(path))
147
148
149class TestGzip(TestCase):
150 """Tests for gzip helpers."""
151
152 def test_gzip_data(self):
153 data = 'foobarbaz\n'
154 compressed = gzip_data(data)
155 fd, path = tempfile.mkstemp()
156 os.write(fd, compressed)
157 os.close(fd)
158 self.assertEqual(data, gzip.open(path, 'r').read())
159
160 def test_gunzip_data(self):
161 data = 'foobarbaz\n'
162 compressed = gzip_data(data)
163 self.assertEqual(data, gunzip_data(compressed))
164
165
166class RequestHelpers:
167
168 def make_trunk(self, parent_url='http://example.com/bzr/trunk'):
169 """Make a trunk branch suitable for use with `Request`.
170
171 `Request` expects to be given a path to a working tree that has a
172 branch with a configured parent URL, so this helper returns such a
173 working tree.
174 """
175 nick = parent_url.strip('/').split('/')[-1]
176 tree = self.make_branch_and_tree(nick)
177 tree.branch.set_parent(parent_url)
178 return tree
179
180 def make_request(self, branch_url=None, revno=None,
181 trunk=None, sourcecode_path=None,
182 emails=None, pqm_message=None):
183 """Make a request to test, specifying only things we care about.
184
185 Note that the returned request object will not ever send email, but
186 will instead log "sent" emails to `request.emails_sent`.
187 """
188 if trunk is None:
189 trunk = self.make_trunk()
190 if sourcecode_path is None:
191 sourcecode_path = self.make_sourcecode(
192 [('a', 'http://example.com/bzr/a', 2),
193 ('b', 'http://example.com/bzr/b', 3),
194 ('c', 'http://example.com/bzr/c', 5)])
195 request = Request(
196 branch_url, revno, trunk.basedir, sourcecode_path, emails,
197 pqm_message)
198 request.emails_sent = []
199 request._send_email = request.emails_sent.append
200 return request
201
202 def make_sourcecode(self, branches):
203 """Make a sourcecode directory with sample branches.
204
205 :param branches: A list of (name, parent_url, revno) tuples.
206 :return: The path to the sourcecode directory.
207 """
208 self.build_tree(['sourcecode/'])
209 for name, parent_url, revno in branches:
210 tree = self.make_branch_and_tree('sourcecode/%s' % (name,))
211 tree.branch.set_parent(parent_url)
212 for i in range(revno):
213 tree.commit(message=str(i))
214 return 'sourcecode/'
215
216 def make_logger(self, request=None, echo_to_stdout=False):
217 if request is None:
218 request = self.make_request()
219 return WebTestLogger(
220 'full.log', 'summary.log', 'index.html', request, echo_to_stdout)
221
222
223class TestRequest(TestCaseWithTransport, RequestHelpers):
224 """Tests for `Request`."""
225
226 def test_doesnt_want_email(self):
227 # If no email addresses were provided, then the user does not want to
228 # receive email.
229 req = self.make_request()
230 self.assertEqual(False, req.wants_email)
231
232 def test_wants_email(self):
233 # If some email addresses were provided, then the user wants to
234 # receive email.
235 req = self.make_request(emails=['foo@example.com'])
236 self.assertEqual(True, req.wants_email)
237
238 def test_get_trunk_details(self):
239 parent = 'http://example.com/bzr/branch'
240 tree = self.make_trunk(parent)
241 req = self.make_request(trunk=tree)
242 self.assertEqual(
243 (parent, tree.branch.revno()), req.get_trunk_details())
244
245 def test_get_branch_details_no_commits(self):
246 req = self.make_request(trunk=self.make_trunk())
247 self.assertEqual(None, req.get_branch_details())
248
249 def test_get_branch_details_no_merge(self):
250 tree = self.make_trunk()
251 tree.commit(message='foo')
252 req = self.make_request(trunk=tree)
253 self.assertEqual(None, req.get_branch_details())
254
255 def test_get_branch_details_merge(self):
256 tree = self.make_trunk()
257 # Fake a merge, giving silly revision ids.
258 tree.add_pending_merge('foo', 'bar')
259 req = self.make_request(
260 branch_url='https://example.com/bzr/thing', revno=42, trunk=tree)
261 self.assertEqual(
262 ('https://example.com/bzr/thing', 42), req.get_branch_details())
263
264 def test_get_nick_trunk_only(self):
265 tree = self.make_trunk(parent_url='http://example.com/bzr/db-devel')
266 req = self.make_request(trunk=tree)
267 self.assertEqual('db-devel', req.get_nick())
268
269 def test_get_nick_merge(self):
270 tree = self.make_trunk()
271 # Fake a merge, giving silly revision ids.
272 tree.add_pending_merge('foo', 'bar')
273 req = self.make_request(
274 branch_url='https://example.com/bzr/thing', revno=42, trunk=tree)
275 self.assertEqual('thing', req.get_nick())
276
277 def test_get_merge_description_trunk_only(self):
278 tree = self.make_trunk(parent_url='http://example.com/bzr/db-devel')
279 req = self.make_request(trunk=tree)
280 self.assertEqual('db-devel', req.get_merge_description())
281
282 def test_get_merge_description_merge(self):
283 tree = self.make_trunk(parent_url='http://example.com/bzr/db-devel/')
284 tree.add_pending_merge('foo', 'bar')
285 req = self.make_request(
286 branch_url='https://example.com/bzr/thing', revno=42, trunk=tree)
287 self.assertEqual('thing => db-devel', req.get_merge_description())
288
289 def test_get_summary_commit(self):
290 # The summary commit message is the last commit message of the branch
291 # we're merging in.
292 trunk = self.make_trunk()
293 trunk.commit(message="a starting point")
294 thing_bzrdir = trunk.branch.bzrdir.sprout('thing')
295 thing = thing_bzrdir.open_workingtree()
296 thing.commit(message="a new thing")
297 trunk.merge_from_branch(thing.branch)
298 req = self.make_request(
299 branch_url='https://example.com/bzr/thing',
300 revno=thing.branch.revno(),
301 trunk=trunk)
302 self.assertEqual("a new thing", req.get_summary_commit())
303
304 def test_iter_dependency_branches(self):
305 # iter_dependency_branches yields a list of branches in the sourcecode
306 # directory, along with their parent URLs and their revnos.
307 sourcecode_branches = [
308 ('b', 'http://example.com/parent-b', 3),
309 ('a', 'http://example.com/parent-a', 2),
310 ('c', 'http://example.com/parent-c', 5),
311 ]
312 sourcecode_path = self.make_sourcecode(sourcecode_branches)
313 self.build_tree(
314 ['%s/not-a-branch/' % sourcecode_path,
315 '%s/just-a-file' % sourcecode_path])
316 req = self.make_request(sourcecode_path=sourcecode_path)
317 branches = list(req.iter_dependency_branches())
318 self.assertEqual(sorted(sourcecode_branches), branches)
319
320 def test_submit_to_pqm_no_message(self):
321 # If there's no PQM message, then 'submit_to_pqm' returns None.
322 req = self.make_request(pqm_message=None)
323 subject = req.submit_to_pqm(successful=True)
324 self.assertIs(None, subject)
325
326 def test_submit_to_pqm_no_message_doesnt_send(self):
327 # If there's no PQM message, then 'submit_to_pqm' returns None.
328 req = self.make_request(pqm_message=None)
329 req.submit_to_pqm(successful=True)
330 self.assertEqual([], req.emails_sent)
331
332 def test_submit_to_pqm_unsuccessful(self):
333 # submit_to_pqm returns the subject of the PQM mail even if it's
334 # handling a failed test run.
335 message = {'Subject:': 'My PQM message'}
336 req = self.make_request(pqm_message=message)
337 subject = req.submit_to_pqm(successful=False)
338 self.assertIs(message.get('Subject'), subject)
339
340 def test_submit_to_pqm_unsuccessful_no_email(self):
341 # submit_to_pqm doesn't send any email if the run was unsuccessful.
342 message = {'Subject:': 'My PQM message'}
343 req = self.make_request(pqm_message=message)
344 req.submit_to_pqm(successful=False)
345 self.assertEqual([], req.emails_sent)
346
347 def test_submit_to_pqm_successful(self):
348 # submit_to_pqm returns the subject of the PQM mail.
349 message = {'Subject:': 'My PQM message'}
350 req = self.make_request(pqm_message=message)
351 subject = req.submit_to_pqm(successful=True)
352 self.assertIs(message.get('Subject'), subject)
353 self.assertEqual([message], req.emails_sent)
354
355 def test_report_email_subject_success(self):
356 req = self.make_request(emails=['foo@example.com'])
357 email = req._build_report_email(True, 'foo', 'gobbledygook')
358 self.assertEqual('Test results: SUCCESS', email['Subject'])
359
360 def test_report_email_subject_failure(self):
361 req = self.make_request(emails=['foo@example.com'])
362 email = req._build_report_email(False, 'foo', 'gobbledygook')
363 self.assertEqual('Test results: FAILURE', email['Subject'])
364
365 def test_report_email_recipients(self):
366 req = self.make_request(emails=['foo@example.com', 'bar@example.com'])
367 email = req._build_report_email(False, 'foo', 'gobbledygook')
368 self.assertEqual('foo@example.com, bar@example.com', email['To'])
369
370 def test_report_email_sender(self):
371 req = self.make_request(emails=['foo@example.com'])
372 email = req._build_report_email(False, 'foo', 'gobbledygook')
373 self.assertEqual(GlobalConfig().username(), email['From'])
374
375 def test_report_email_body(self):
376 req = self.make_request(emails=['foo@example.com'])
377 email = req._build_report_email(False, 'foo', 'gobbledygook')
378 [body, attachment] = email.get_payload()
379 self.assertIsInstance(body, MIMEText)
380 self.assertEqual('inline', body['Content-Disposition'])
381 self.assertEqual('text/plain; charset="utf-8"', body['Content-Type'])
382 self.assertEqual("foo", body.get_payload())
383
384 def test_report_email_attachment(self):
385 req = self.make_request(emails=['foo@example.com'])
386 email = req._build_report_email(False, "foo", "gobbledygook")
387 [body, attachment] = email.get_payload()
388 self.assertIsInstance(attachment, MIMEApplication)
389 self.assertEqual('application/x-gzip', attachment['Content-Type'])
390 self.assertEqual(
391 'attachment; filename="%s.log.gz"' % req.get_nick(),
392 attachment['Content-Disposition'])
393 self.assertEqual(
394 "gobbledygook", attachment.get_payload().decode('base64'))
395
396 def test_send_report_email_sends_email(self):
397 req = self.make_request(emails=['foo@example.com'])
398 expected = req._build_report_email(False, "foo", "gobbledygook")
399 req.send_report_email(False, "foo", "gobbledygook")
400 [observed] = req.emails_sent
401 # The standard library sucks. None of the MIME objects have __eq__
402 # implementations.
403 for expected_part, observed_part in izip(
404 expected.walk(), observed.walk()):
405 self.assertEqual(type(expected_part), type(observed_part))
406 self.assertEqual(expected_part.items(), observed_part.items())
407 self.assertEqual(
408 expected_part.is_multipart(), observed_part.is_multipart())
409 if not expected_part.is_multipart():
410 self.assertEqual(
411 expected_part.get_payload(), observed_part.get_payload())
412
413
414class TestWebTestLogger(TestCaseWithTransport, RequestHelpers):
415
416 def patch(self, obj, name, value):
417 orig = getattr(obj, name)
418 setattr(obj, name, value)
419 self.addCleanup(setattr, obj, name, orig)
420 return orig
421
422 def test_make_in_directory(self):
423 # WebTestLogger.make_in_directory constructs a logger that writes to a
424 # bunch of specific files in a directory.
425 self.build_tree(['www/'])
426 request = self.make_request()
427 logger = WebTestLogger.make_in_directory('www', request, False)
428 # A method on logger that writes to _everything_.
429 logger.prepare()
430 self.assertEqual(
431 logger.get_summary_contents(), open('www/summary.log').read())
432 self.assertEqual(
433 logger.get_full_log_contents(),
434 open('www/current_test.log').read())
435 self.assertEqual(
436 logger.get_index_contents(), open('www/index.html').read())
437
438 def test_initial_full_log(self):
439 # Initially, the full log has nothing in it.
440 logger = self.make_logger()
441 self.assertEqual('', logger.get_full_log_contents())
442
443 def test_initial_summary_contents(self):
444 # Initially, the summary log has nothing in it.
445 logger = self.make_logger()
446 self.assertEqual('', logger.get_summary_contents())
447
448 def test_got_line_no_echo(self):
449 # got_line forwards the line to the full log, but does not forward to
450 # stdout if echo_to_stdout is False.
451 stdout = StringIO()
452 self.patch(sys, 'stdout', stdout)
453 logger = self.make_logger(echo_to_stdout=False)
454 logger.got_line("output from script\n")
455 self.assertEqual(
456 "output from script\n", logger.get_full_log_contents())
457 self.assertEqual("", stdout.getvalue())
458
459 def test_got_line_echo(self):
460 # got_line forwards the line to the full log, and to stdout if
461 # echo_to_stdout is True.
462 stdout = StringIO()
463 self.patch(sys, 'stdout', stdout)
464 logger = self.make_logger(echo_to_stdout=True)
465 logger.got_line("output from script\n")
466 self.assertEqual(
467 "output from script\n", logger.get_full_log_contents())
468 self.assertEqual("output from script\n", stdout.getvalue())
469
470 def test_write_line(self):
471 # write_line writes a line to both the full log and the summary log.
472 logger = self.make_logger()
473 logger.write_line('foo')
474 self.assertEqual('foo\n', logger.get_full_log_contents())
475 self.assertEqual('foo\n', logger.get_summary_contents())
476
477 def test_error_in_testrunner(self):
478 # error_in_testrunner logs the traceback to the summary log in a very
479 # prominent way.
480 try:
481 1/0
482 except ZeroDivisionError:
483 exc_info = sys.exc_info()
484 stack = ''.join(traceback.format_exception(*exc_info))
485 logger = self.make_logger()
486 logger.error_in_testrunner(exc_info)
487 self.assertEqual(
488 "\n\nERROR IN TESTRUNNER\n\n%s" % (stack,),
489 logger.get_summary_contents())
490
491
492class TestDaemonizationInteraction(TestCaseWithTransport, RequestHelpers):
493
494 script_file = 'remote_daemonization_test.py'
495
496 def run_script(self, script_file, pidfile, directory, logfile):
497 path = os.path.join(os.path.dirname(__file__), script_file)
498 popen = subprocess.Popen(
499 [sys.executable, path, pidfile, directory, logfile],
500 stdout=subprocess.PIPE, stderr=subprocess.PIPE)
501 stdout, stderr = popen.communicate()
502 self.assertEqual((0, '', ''), (popen.returncode, stdout, stderr))
503 # Make sure the daemon is finished doing its thing.
504 while os.path.exists(pidfile):
505 time.sleep(0.01)
506 # If anything was written to 'logfile' while the script was running,
507 # we want it to appear in our test errors. This way, stack traces in
508 # the script are visible to us as test runners.
509 if os.path.exists(logfile):
510 content_type = ContentType("text", "plain", {"charset": "utf8"})
511 content = Content(content_type, open(logfile).read)
512 self.addDetail('logfile', content)
513
514 def test_daemonization(self):
515 # Daemonizing something can do funny things to its behavior. This test
516 # runs a script that's very similar to remote.py but only does
517 # "logger.prepare".
518 pidfile = "%s.pid" % self.id()
519 directory = 'www'
520 logfile = "%s.log" % self.id()
521 self.run_script(self.script_file, pidfile, directory, logfile)
522 # Check that the output from the daemonized version matches the output
523 # from a normal version. Only checking one of the files, since this is
524 # more of a smoke test than a correctness test.
525 logger = self.make_logger()
526 logger.prepare()
527 expected_summary = logger.get_summary_contents()
528 observed_summary = open(os.path.join(directory, 'summary.log')).read()
529 # The first line contains a timestamp, so we ignore it.
530 self.assertEqual(
531 expected_summary.splitlines()[1:],
532 observed_summary.splitlines()[1:])
533
534
535class TestResultHandling(TestCaseWithTransport, RequestHelpers):
536 """Tests for how we handle the result at the end of the test suite."""
537
538 def get_body_text(self, email):
539 return email.get_payload()[0].get_payload()
540
541 def test_success_no_emails(self):
542 request = self.make_request(emails=[])
543 logger = self.make_logger(request=request)
544 logger.got_result(True)
545 self.assertEqual([], request.emails_sent)
546
547 def test_failure_no_emails(self):
548 request = self.make_request(emails=[])
549 logger = self.make_logger(request=request)
550 logger.got_result(False)
551 self.assertEqual([], request.emails_sent)
552
553 def test_submits_to_pqm_on_success(self):
554 message = {'Subject': 'foo'}
555 request = self.make_request(emails=[], pqm_message=message)
556 logger = self.make_logger(request=request)
557 logger.got_result(True)
558 self.assertEqual([message], request.emails_sent)
559
560 def test_records_pqm_submission_in_email(self):
561 message = {'Subject': 'foo'}
562 request = self.make_request(
563 emails=['foo@example.com'], pqm_message=message)
564 logger = self.make_logger(request=request)
565 logger.got_result(True)
566 [pqm_message, user_message] = request.emails_sent
567 self.assertEqual(message, pqm_message)
568 self.assertIn(
569 'SUBMITTED TO PQM:\n%s' % (message['Subject'],),
570 self.get_body_text(user_message))
571
572 def test_doesnt_submit_to_pqm_no_failure(self):
573 message = {'Subject': 'foo'}
574 request = self.make_request(emails=[], pqm_message=message)
575 logger = self.make_logger(request=request)
576 logger.got_result(False)
577 self.assertEqual([], request.emails_sent)
578
579 def test_records_non_pqm_submission_in_email(self):
580 message = {'Subject': 'foo'}
581 request = self.make_request(
582 emails=['foo@example.com'], pqm_message=message)
583 logger = self.make_logger(request=request)
584 logger.got_result(False)
585 [user_message] = request.emails_sent
586 self.assertIn(
587 '**NOT** submitted to PQM:\n%s' % (message['Subject'],),
588 self.get_body_text(user_message))
589
590 def test_email_refers_to_attached_log(self):
591 request = self.make_request(emails=['foo@example.com'])
592 logger = self.make_logger(request=request)
593 logger.got_result(False)
594 [user_message] = request.emails_sent
595 self.assertIn(
596 '(See the attached file for the complete log)\n',
597 self.get_body_text(user_message))
598
599 def test_email_body_is_summary(self):
600 # The body of the email sent to the user is the summary file.
601 # XXX: JonathanLange 2010-08-17: We actually want to change this
602 # behaviour so that the summary log stays roughly the same as it is
603 # now and can vary independently from the contents of the sent
604 # email. We probably just want the contents of the email to be a list
605 # of failing tests.
606 request = self.make_request(emails=['foo@example.com'])
607 logger = self.make_logger(request=request)
608 logger.get_summary_stream().write('bar\nbaz\nqux\n')
609 logger.got_result(False)
610 [user_message] = request.emails_sent
611 self.assertEqual(
612 'bar\nbaz\nqux\n\n(See the attached file for the complete log)\n',
613 self.get_body_text(user_message))
614
615 def test_gzip_of_full_log_attached(self):
616 # The full log is attached to the email.
617 request = self.make_request(emails=['foo@example.com'])
618 logger = self.make_logger(request=request)
619 logger.got_line("output from test process\n")
620 logger.got_line("more output\n")
621 logger.got_result(False)
622 [user_message] = request.emails_sent
623 [body, attachment] = user_message.get_payload()
624 self.assertEqual('application/x-gzip', attachment['Content-Type'])
625 self.assertEqual(
626 'attachment; filename="%s.log.gz"' % request.get_nick(),
627 attachment['Content-Disposition'])
628 attachment_contents = attachment.get_payload().decode('base64')
629 uncompressed = gunzip_data(attachment_contents)
630 self.assertEqual(
631 "output from test process\nmore output\n", uncompressed)
632
70633
71def test_suite():634def test_suite():
72 return unittest.TestLoader().loadTestsFromName(__name__)635 return unittest.TestLoader().loadTestsFromName(__name__)