Code review comment for lp:~jameinel/bzr/2.3-filter-tests

we did two simultaneous things a while back:
 - turned off -1, so that we get *all* failures
 - turned on subunit, so that the failures are machine driven

The emails are meant to be compressed, so you should in principle be
able get it reliably.

If we're actually running into mailer limits, we can do a few things:
 - turn off subunit (will reduce some of the overhead - lots in a
success case, but in a it-all-goes-pear-shaped failure, you'll still
hit the limit and whatever undiagnosed and unfixed issue is biting you
now, will bite you then.
 - diagnose and fix the mailer path issue so that you receive the mails reliably
 - stop using email to send the results
 - filter the stream at source so success and other results that you
consider uninteresting are not included in the 'raw' stream. Note that
again, this will *leave the failure mode in place so when it goes bad
you will stop getting the responses*

Now, I'm not directly experiencing this anymore, so you need to do
what makes sense to you.

If subunit's API is out of date and messing up the reason formatting,
I'll happily fix it - I'm going to go peek and see if its addSkip is
stale right after mailing this.

To stop using email to send the results, we either need to design a
new thing and change PQM to do it, or reenable the LP API support,
which Tim requested we disable.

We previously *haven't* filtered at source as a defensive measure
against bugs. Python 2 is riddled with default-encoding pitfalls and
they made subunit + bzr very unreliable at one point. Possibly totally
fixed thanks to Martin[gz].

Personally, I would not take any action that leaves the problem fallow
waiting for a serious test failure to provoke it: that just means that
when you need it most it will fail. However, as I say above: you're
using this system, I'm rarely using it : do what makes sense to you. I
do strongly suggest that you change things in PQM itself if you want
filtering, rather than bzr. That will at least preserve the detailed
output for babune if you do need it in future.

As for the value of logs on success, xfail, skip etc : *if* they pass
incorrectly, I think the log will be invaluable, but you won't need it
till you need it.

-Rob