Launchpad itself

Merge lp:~jml/launchpad/list-ec2-test-runs-721784 into lp:launchpad

  1. list-ec2-test-runs-721784
  2. Merge into devel
Proposed by Jonathan Lange
Status: Merged
Approved by: Leonard Richardson
Approved revision: no longer in the source branch.
Merged at revision: 12453
Proposed branch: lp:~jml/launchpad/list-ec2-test-runs-721784
Merge into: lp:launchpad
Diff against target: 526 lines (+238/-77)
3 files modified
lib/devscripts/ec2test/builtins.py (+111/-1)
lib/devscripts/ec2test/remote.py (+45/-35)
lib/devscripts/ec2test/tests/test_remote.py (+82/-41)
To merge this branch: bzr merge lp:~jml/launchpad/list-ec2-test-runs-721784
Reviewer Review Type Date Requested Status
Leonard Richardson (community) Approve
Review via email: mp+50537@code.launchpad.net

Commit message

[no-qa] [r=leonardr][bug=721784] Add ec2 list command to list running ec2 instances and test status

Description of the change

This branch adds a new command 'ec2 list', which lists your running ec2 instances, tells you if they are currently passing, and how long they've been running for.

The implementation approach is to have the remote instance publish a JSON file, and have 'ec2 list' scan all instances for the JSON file. Still have some details to work out re exact output.

To post a comment you must log in.
Revision history for this message
Jonathan Lange (jml) wrote :

Got something worth looking out now. It's not perfect, but I personally think it's good enough to land.

Revision history for this message
Leonard Richardson (leonardr) wrote :

This is a good start. I'd like to see some tests of cmd_list, especially format_summary and format_instance, but that can wait for a follow-up.

I don't think you need to call .replace(tzinfo=UTC) on datetime.utcnow(), but it doesn't hurt.

I think using 'success: True' to describe a run that's been successful *so far* might give people the wrong impression? Is there an easy way to distinguish between the case when you list a run that's shutting down after a complete success and a run that just hasn't failed yet?

review: Approve
Revision history for this message
Jonathan Lange (jml) wrote :

Thanks for the review.

You do have to replace the tzinfo from utcnow. It returns a naive timestamp.

I can say 'failed-yet' in the metadata, still format the output as [OK], and make this clear in the documentation for the command.

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== modified file 'lib/devscripts/ec2test/builtins.py'
--- lib/devscripts/ec2test/builtins.py 2010-11-29 21:32:06 +0000
+++ lib/devscripts/ec2test/builtins.py 2011-02-23 11:43:52 +0000
@@ -6,6 +6,7 @@
6__metaclass__ = type6__metaclass__ = type
7__all__ = []7__all__ = []
88
9from datetime import datetime
9import os10import os
10import pdb11import pdb
11import socket12import socket
@@ -13,12 +14,20 @@
1314
14from bzrlib.bzrdir import BzrDir15from bzrlib.bzrdir import BzrDir
15from bzrlib.commands import Command16from bzrlib.commands import Command
16from bzrlib.errors import BzrCommandError17from bzrlib.errors import (
18 BzrCommandError,
19 ConnectionError,
20 NoSuchFile,
21 )
17from bzrlib.help import help_commands22from bzrlib.help import help_commands
18from bzrlib.option import (23from bzrlib.option import (
19 ListOption,24 ListOption,
20 Option,25 Option,
21 )26 )
27from bzrlib.transport import get_transport
28from pytz import UTC
29import simplejson
30
22from devscripts import get_launchpad_root31from devscripts import get_launchpad_root
23from devscripts.ec2test.account import VALID_AMI_OWNERS32from devscripts.ec2test.account import VALID_AMI_OWNERS
24from devscripts.ec2test.credentials import EC2Credentials33from devscripts.ec2test.credentials import EC2Credentials
@@ -652,6 +661,107 @@
652 VALID_AMI_OWNERS.get(image.ownerId, "unknown")))661 VALID_AMI_OWNERS.get(image.ownerId, "unknown")))
653662
654663
664class cmd_list(EC2Command):
665 """List all your current EC2 test runs.
666
667 If an instance is publishing an 'info.json' file with 'description' and
668 'failed-yet' fields, this command will list that instance, whether it has
669 failed the test run and how long it has been up for.
670
671 [FAILED] means that the has been a failing test. [OK] means that the test
672 run has had no failures yet, it's not a guarantee of a successful run.
673 """
674
675 aliases = ["ls"]
676
677 takes_options = [
678 Option('show-urls',
679 help="Include more information about each instance"),
680 Option('all', short_name='a',
681 help="Show all instances, not just ones with ec2test data."),
682 ]
683
684 def iter_instances(self, account):
685 """Iterate through all instances in 'account'."""
686 for reservation in account.conn.get_all_instances():
687 for instance in reservation.instances:
688 yield instance
689
690 def get_uptime(self, instance):
691 """How long has 'instance' been running?"""
692 expected_format = '%Y-%m-%dT%H:%M:%S.000Z'
693 launch_time = datetime.strptime(instance.launch_time, expected_format)
694 return (
695 datetime.utcnow().replace(tzinfo=UTC)
696 - launch_time.replace(tzinfo=UTC))
697
698 def get_http_url(self, instance):
699 hostname = instance.public_dns_name
700 if not hostname:
701 return
702 return 'http://%s/' % (hostname,)
703
704 def get_ec2test_info(self, instance):
705 """Load the ec2test-specific information published by 'instance'."""
706 url = self.get_http_url(instance)
707 if url is None:
708 return
709 try:
710 json = get_transport(url).get_bytes('info.json')
711 except (ConnectionError, NoSuchFile):
712 # Probably not an ec2test instance, or not ready yet.
713 return None
714 return simplejson.loads(json)
715
716 def format_instance(self, instance, data, verbose):
717 """Format 'instance' for display.
718
719 :param instance: The EC2 instance to display.
720 :param data: Launchpad-specific data.
721 :param verbose: Whether we want verbose output.
722 """
723 uptime = self.get_uptime(instance)
724 if data is None:
725 description = instance.id
726 current_status = 'unknown '
727 else:
728 description = data['description']
729 if data['failed-yet']:
730 current_status = '[FAILED]'
731 else:
732 current_status = '[OK] '
733 output = '%s %s (up for %s)' % (description, current_status, uptime)
734 if verbose:
735 url = self.get_http_url(instance)
736 if url is None:
737 url = "No web service"
738 output += '\n %s' % (url,)
739 return output
740
741 def format_summary(self, by_state):
742 return ', '.join(
743 ': '.join((state, str(num)))
744 for (state, num) in sorted(list(by_state.items())))
745
746 def run(self, show_urls=False, all=False):
747 credentials = EC2Credentials.load_from_file()
748 session_name = EC2SessionName.make(EC2TestRunner.name)
749 account = credentials.connect(session_name)
750 instances = list(self.iter_instances(account))
751 if len(instances) == 0:
752 print "No instances running."
753 return
754
755 by_state = {}
756 for instance in instances:
757 by_state[instance.state] = by_state.get(instance.state, 0) + 1
758 data = self.get_ec2test_info(instance)
759 if data is None and not all:
760 continue
761 print self.format_instance(instance, data, show_urls)
762 print 'Summary: %s' % (self.format_summary(by_state),)
763
764
655class cmd_help(EC2Command):765class cmd_help(EC2Command):
656 """Show general help or help for a command."""766 """Show general help or help for a command."""
657767
658768
=== modified file 'lib/devscripts/ec2test/remote.py'
--- lib/devscripts/ec2test/remote.py 2010-10-26 15:47:24 +0000
+++ lib/devscripts/ec2test/remote.py 2011-02-23 11:43:52 +0000
@@ -10,7 +10,7 @@
10 "test merging foo-bar-bug-12345 into db-devel").10 "test merging foo-bar-bug-12345 into db-devel").
1111
12 * `LaunchpadTester` knows how to actually run the tests and gather the12 * `LaunchpadTester` knows how to actually run the tests and gather the
13 results. It uses `SummaryResult` and `FlagFallStream` to do so.13 results. It uses `SummaryResult` to do so.
1414
15 * `WebTestLogger` knows how to display the results to the user, and is given15 * `WebTestLogger` knows how to display the results to the user, and is given
16 the responsibility of handling the results that `LaunchpadTester` gathers.16 the responsibility of handling the results that `LaunchpadTester` gathers.
@@ -34,9 +34,10 @@
34import time34import time
35import traceback35import traceback
36import unittest36import unittest
37
38from xml.sax.saxutils import escape37from xml.sax.saxutils import escape
3938
39import simplejson
40
40import bzrlib.branch41import bzrlib.branch
41import bzrlib.config42import bzrlib.config
42import bzrlib.errors43import bzrlib.errors
@@ -47,6 +48,8 @@
4748
48import subunit49import subunit
4950
51from testtools import MultiTestResult
52
5053
51class NonZeroExitCode(Exception):54class NonZeroExitCode(Exception):
52 """Raised when the child process exits with a non-zero exit code."""55 """Raised when the child process exits with a non-zero exit code."""
@@ -94,34 +97,19 @@
94 self.stream.flush()97 self.stream.flush()
9598
9699
97class FlagFallStream:100class FailureUpdateResult(unittest.TestResult):
98 """Wrapper around a stream that only starts forwarding after a flagfall.101
99 """102 def __init__(self, logger):
100103 super(FailureUpdateResult, self).__init__()
101 def __init__(self, stream, flag):104 self._logger = logger
102 """Construct a `FlagFallStream` that wraps 'stream'.105
103106 def addError(self, *args, **kwargs):
104 :param stream: A stream, a file-like object.107 super(FailureUpdateResult, self).addError(*args, **kwargs)
105 :param flag: A string that needs to be written to this stream before108 self._logger.got_failure()
106 we start forwarding the output.109
107 """110 def addFailure(self, *args, **kwargs):
108 self._stream = stream111 super(FailureUpdateResult, self).addFailure(*args, **kwargs)
109 self._flag = flag112 self._logger.got_failure()
110 self._flag_fallen = False
111
112 def write(self, bytes):
113 if self._flag_fallen:
114 self._stream.write(bytes)
115 else:
116 index = bytes.find(self._flag)
117 if index == -1:
118 return
119 else:
120 self._stream.write(bytes[index:])
121 self._flag_fallen = True
122
123 def flush(self):
124 self._stream.flush()
125113
126114
127class EC2Runner:115class EC2Runner:
@@ -293,7 +281,9 @@
293 def _gather_test_output(self, input_stream, logger):281 def _gather_test_output(self, input_stream, logger):
294 """Write the testrunner output to the logs."""282 """Write the testrunner output to the logs."""
295 summary_stream = logger.get_summary_stream()283 summary_stream = logger.get_summary_stream()
296 result = SummaryResult(summary_stream)284 result = MultiTestResult(
285 SummaryResult(summary_stream),
286 FailureUpdateResult(logger))
297 subunit_server = subunit.TestProtocolServer(result, summary_stream)287 subunit_server = subunit.TestProtocolServer(result, summary_stream)
298 for line in input_stream:288 for line in input_stream:
299 subunit_server.lineReceived(line)289 subunit_server.lineReceived(line)
@@ -302,6 +292,8 @@
302 return result292 return result
303293
304294
295# XXX: Publish a JSON file that includes the relevant details from this
296# request.
305class Request:297class Request:
306 """A request to have a branch tested and maybe landed."""298 """A request to have a branch tested and maybe landed."""
307299
@@ -433,9 +425,6 @@
433 we're just running tests for a trunk branch without merging return425 we're just running tests for a trunk branch without merging return
434 '$TRUNK_NICK'.426 '$TRUNK_NICK'.
435 """427 """
436 # XXX: JonathanLange 2010-08-17: Not actually used yet. I think it
437 # would be a great thing to have in the subject of the emails we
438 # receive.
439 source = self.get_source_details()428 source = self.get_source_details()
440 if not source:429 if not source:
441 return '%s r%s' % (self.get_nick(), self.get_revno())430 return '%s r%s' % (self.get_nick(), self.get_revno())
@@ -532,7 +521,11 @@
532521
533522
534class WebTestLogger:523class WebTestLogger:
535 """Logs test output to disk and a simple web page."""524 """Logs test output to disk and a simple web page.
525
526 :ivar successful: Whether the logger has received only successful input up
527 until now.
528 """
536529
537 def __init__(self, full_log_filename, summary_filename, index_filename,530 def __init__(self, full_log_filename, summary_filename, index_filename,
538 request, echo_to_stdout):531 request, echo_to_stdout):
@@ -556,11 +549,14 @@
556 self._full_log_filename = full_log_filename549 self._full_log_filename = full_log_filename
557 self._summary_filename = summary_filename550 self._summary_filename = summary_filename
558 self._index_filename = index_filename551 self._index_filename = index_filename
552 self._info_json = os.path.join(
553 os.path.dirname(index_filename), 'info.json')
559 self._request = request554 self._request = request
560 self._echo_to_stdout = echo_to_stdout555 self._echo_to_stdout = echo_to_stdout
561 # Actually set by prepare(), but setting to a dummy value to make556 # Actually set by prepare(), but setting to a dummy value to make
562 # testing easier.557 # testing easier.
563 self._start_time = datetime.datetime.utcnow()558 self._start_time = datetime.datetime.utcnow()
559 self.successful = True
564560
565 @classmethod561 @classmethod
566 def make_in_directory(cls, www_dir, request, echo_to_stdout):562 def make_in_directory(cls, www_dir, request, echo_to_stdout):
@@ -628,6 +624,11 @@
628 if e.errno == errno.ENOENT:624 if e.errno == errno.ENOENT:
629 return ''625 return ''
630626
627 def got_failure(self):
628 """Called when we receive word that a test has failed."""
629 self.successful = False
630 self._dump_json()
631
631 def got_result(self, result):632 def got_result(self, result):
632 """The tests are done and the results are known."""633 """The tests are done and the results are known."""
633 self._end_time = datetime.datetime.utcnow()634 self._end_time = datetime.datetime.utcnow()
@@ -666,12 +667,21 @@
666 """Write to the summary and full log file with a newline."""667 """Write to the summary and full log file with a newline."""
667 self._write(msg + '\n')668 self._write(msg + '\n')
668669
670 def _dump_json(self):
671 fd = open(self._info_json, 'w')
672 simplejson.dump(
673 {'description': self._request.get_merge_description(),
674 'failed-yet': not self.successful,
675 }, fd)
676 fd.close()
677
669 def prepare(self):678 def prepare(self):
670 """Prepares the log files on disk.679 """Prepares the log files on disk.
671680
672 Writes three log files: the raw output log, the filtered "summary"681 Writes three log files: the raw output log, the filtered "summary"
673 log file, and a HTML index page summarizing the test run paramters.682 log file, and a HTML index page summarizing the test run paramters.
674 """683 """
684 self._dump_json()
675 # XXX: JonathanLange 2010-07-18: Mostly untested.685 # XXX: JonathanLange 2010-07-18: Mostly untested.
676 log = self.write_line686 log = self.write_line
677687
678688
=== modified file 'lib/devscripts/ec2test/tests/test_remote.py'
--- lib/devscripts/ec2test/tests/test_remote.py 2010-10-06 11:46:51 +0000
+++ lib/devscripts/ec2test/tests/test_remote.py 2011-02-23 11:43:52 +0000
@@ -20,6 +20,8 @@
20import traceback20import traceback
21import unittest21import unittest
2222
23import simplejson
24
23from bzrlib.config import GlobalConfig25from bzrlib.config import GlobalConfig
24from bzrlib.tests import TestCaseWithTransport26from bzrlib.tests import TestCaseWithTransport
2527
@@ -30,7 +32,7 @@
3032
31from devscripts.ec2test.remote import (33from devscripts.ec2test.remote import (
32 EC2Runner,34 EC2Runner,
33 FlagFallStream,35 FailureUpdateResult,
34 gunzip_data,36 gunzip_data,
35 gzip_data,37 gzip_data,
36 LaunchpadTester,38 LaunchpadTester,
@@ -122,39 +124,6 @@
122 'full.log', 'summary.log', 'index.html', request, echo_to_stdout)124 'full.log', 'summary.log', 'index.html', request, echo_to_stdout)
123125
124126
125class TestFlagFallStream(TestCase):
126 """Tests for `FlagFallStream`."""
127
128 def test_doesnt_write_before_flag(self):
129 # A FlagFallStream does not forward any writes before it sees the
130 # 'flag'.
131 stream = StringIO()
132 flag = self.getUniqueString('flag')
133 flagfall = FlagFallStream(stream, flag)
134 flagfall.write('foo')
135 flagfall.flush()
136 self.assertEqual('', stream.getvalue())
137
138 def test_writes_after_flag(self):
139 # After a FlagFallStream sees the flag, it forwards all writes.
140 stream = StringIO()
141 flag = self.getUniqueString('flag')
142 flagfall = FlagFallStream(stream, flag)
143 flagfall.write('foo')
144 flagfall.write(flag)
145 flagfall.write('bar')
146 self.assertEqual('%sbar' % (flag,), stream.getvalue())
147
148 def test_mixed_write(self):
149 # If a single call to write has pre-flagfall and post-flagfall data in
150 # it, then only the post-flagfall data is forwarded to the stream.
151 stream = StringIO()
152 flag = self.getUniqueString('flag')
153 flagfall = FlagFallStream(stream, flag)
154 flagfall.write('foo%sbar' % (flag,))
155 self.assertEqual('%sbar' % (flag,), stream.getvalue())
156
157
158class TestSummaryResult(TestCase):127class TestSummaryResult(TestCase):
159 """Tests for `SummaryResult`."""128 """Tests for `SummaryResult`."""
160129
@@ -207,6 +176,29 @@
207 self.assertEqual(1, len(flush_calls))176 self.assertEqual(1, len(flush_calls))
208177
209178
179class TestFailureUpdateResult(TestCaseWithTransport, RequestHelpers):
180
181 def makeException(self, factory=None, *args, **kwargs):
182 if factory is None:
183 factory = RuntimeError
184 try:
185 raise factory(*args, **kwargs)
186 except:
187 return sys.exc_info()
188
189 def test_addError_is_unsuccessful(self):
190 logger = self.make_logger()
191 result = FailureUpdateResult(logger)
192 result.addError(self, self.makeException())
193 self.assertEqual(False, logger.successful)
194
195 def test_addFailure_is_unsuccessful(self):
196 logger = self.make_logger()
197 result = FailureUpdateResult(logger)
198 result.addFailure(self, self.makeException(AssertionError))
199 self.assertEqual(False, logger.successful)
200
201
210class FakePopen:202class FakePopen:
211 """Fake Popen object so we don't have to spawn processes in tests."""203 """Fake Popen object so we don't have to spawn processes in tests."""
212204
@@ -267,6 +259,16 @@
267 # Message being sent implies got_result thought it got a success.259 # Message being sent implies got_result thought it got a success.
268 self.assertEqual([message], log)260 self.assertEqual([message], log)
269261
262 def test_failing_test(self):
263 # If LaunchpadTester gets a failing test, then it records that on the
264 # logger.
265 logger = self.make_logger()
266 tester = self.make_tester(logger=logger)
267 output = "test: foo\nerror: foo\n"
268 tester._spawn_test_process = lambda: FakePopen(output, 0)
269 tester.test()
270 self.assertEqual(False, logger.successful)
271
270 def test_error_in_testrunner(self):272 def test_error_in_testrunner(self):
271 # Any exception is raised within LaunchpadTester.test() is an error in273 # Any exception is raised within LaunchpadTester.test() is an error in
272 # the testrunner. When we detect these, we do three things:274 # the testrunner. When we detect these, we do three things:
@@ -526,8 +528,8 @@
526 [body, attachment] = email.get_payload()528 [body, attachment] = email.get_payload()
527 self.assertIsInstance(body, MIMEText)529 self.assertIsInstance(body, MIMEText)
528 self.assertEqual('inline', body['Content-Disposition'])530 self.assertEqual('inline', body['Content-Disposition'])
529 self.assertEqual('text/plain; charset="utf-8"', body['Content-Type'])531 self.assertEqual('text/plain; charset="utf8"', body['Content-Type'])
530 self.assertEqual("foo", body.get_payload())532 self.assertEqual("foo", body.get_payload(decode=True))
531533
532 def test_report_email_attachment(self):534 def test_report_email_attachment(self):
533 req = self.make_request(emails=['foo@example.com'])535 req = self.make_request(emails=['foo@example.com'])
@@ -540,7 +542,7 @@
540 req.get_nick(), req.get_revno()),542 req.get_nick(), req.get_revno()),
541 attachment['Content-Disposition'])543 attachment['Content-Disposition'])
542 self.assertEqual(544 self.assertEqual(
543 "gobbledygook", attachment.get_payload().decode('base64'))545 "gobbledygook", attachment.get_payload(decode=True))
544546
545 def test_send_report_email_sends_email(self):547 def test_send_report_email_sends_email(self):
546 log = []548 log = []
@@ -707,6 +709,43 @@
707 logger = self.make_logger()709 logger = self.make_logger()
708 self.assertEqual('', logger.get_summary_contents())710 self.assertEqual('', logger.get_summary_contents())
709711
712 def test_initial_json(self):
713 self.build_tree(['www/'])
714 request = self.make_request()
715 logger = WebTestLogger.make_in_directory('www', request, False)
716 logger.prepare()
717 self.assertEqual(
718 {'description': request.get_merge_description(),
719 'successful': True,
720 },
721 simplejson.loads(open('www/info.json').read()))
722
723 def test_initial_success(self):
724 # The Logger initially thinks it is successful because there have been
725 # no failures yet.
726 logger = self.make_logger()
727 self.assertEqual(True, logger.successful)
728
729 def test_got_failure_changes_success(self):
730 # Logger.got_failure() tells the logger it is no longer successful.
731 logger = self.make_logger()
732 logger.got_failure()
733 self.assertEqual(False, logger.successful)
734
735 def test_got_failure_updates_json(self):
736 # Logger.got_failure() updates JSON so that interested parties can
737 # determine that it is unsuccessful.
738 self.build_tree(['www/'])
739 request = self.make_request()
740 logger = WebTestLogger.make_in_directory('www', request, False)
741 logger.prepare()
742 logger.got_failure()
743 self.assertEqual(
744 {'description': request.get_merge_description(),
745 'successful': False,
746 },
747 simplejson.loads(open('www/info.json').read()))
748
710 def test_got_line_no_echo(self):749 def test_got_line_no_echo(self):
711 # got_line forwards the line to the full log, but does not forward to750 # got_line forwards the line to the full log, but does not forward to
712 # stdout if echo_to_stdout is False.751 # stdout if echo_to_stdout is False.
@@ -769,7 +808,7 @@
769 [body, attachment] = email.get_payload()808 [body, attachment] = email.get_payload()
770 self.assertIsInstance(body, MIMEText)809 self.assertIsInstance(body, MIMEText)
771 self.assertEqual('inline', body['Content-Disposition'])810 self.assertEqual('inline', body['Content-Disposition'])
772 self.assertEqual('text/plain; charset="utf-8"', body['Content-Type'])811 self.assertEqual('text/plain; charset="utf8"', body['Content-Type'])
773 self.assertEqual(812 self.assertEqual(
774 logger.get_summary_contents(), body.get_payload(decode=True))813 logger.get_summary_contents(), body.get_payload(decode=True))
775 self.assertIsInstance(attachment, MIMEApplication)814 self.assertIsInstance(attachment, MIMEApplication)
@@ -838,7 +877,9 @@
838 self.assertNotEqual([], email_log)877 self.assertNotEqual([], email_log)
839 [tester_msg] = email_log878 [tester_msg] = email_log
840 self.assertEqual('foo@example.com', tester_msg['To'])879 self.assertEqual('foo@example.com', tester_msg['To'])
841 self.assertIn('ZeroDivisionError', str(tester_msg))880 self.assertIn(
881 'ZeroDivisionError',
882 tester_msg.get_payload()[0].get_payload(decode=True))
842883
843884
844class TestDaemonizationInteraction(TestCaseWithTransport, RequestHelpers):885class TestDaemonizationInteraction(TestCaseWithTransport, RequestHelpers):
@@ -888,7 +929,7 @@
888 """Tests for how we handle the result at the end of the test suite."""929 """Tests for how we handle the result at the end of the test suite."""
889930
890 def get_body_text(self, email):931 def get_body_text(self, email):
891 return email.get_payload()[0].get_payload()932 return email.get_payload()[0].get_payload(decode=True)
892933
893 def make_empty_result(self):934 def make_empty_result(self):
894 return TestResult()935 return TestResult()
@@ -989,7 +1030,7 @@
989 self.assertEqual(1030 self.assertEqual(
990 request.format_result(1031 request.format_result(
991 result, logger._start_time, logger._end_time),1032 result, logger._start_time, logger._end_time),
992 self.get_body_text(user_message).decode('quoted-printable'))1033 self.get_body_text(user_message))
9931034
994 def test_gzip_of_full_log_attached(self):1035 def test_gzip_of_full_log_attached(self):
995 # The full log is attached to the email.1036 # The full log is attached to the email.