Merge lp:~fdo.perez/ipython/trunk-dev into lp:ipython/0.11

Small fix to info message when pylab starts.

Fix tab-completion for magics, other completion cleanup and fixes.

Mostly simplifying the code and updating it, no major functionality changes
other than fixing magic completion, which had broken in the 0.11 refactor.

Also added a small test file for completion-specific tests.

Remove redundant decorators and unify into decorators.py

Added test support for better parametric tests and nose fixes.

These are the underlying support files needed by the rest.

Add new testing support machinery with better parametric tests.

Also included are new tools for doctests with ipython syntax.

Update numpy's decorators.py from upstream.

Fix quitting: now, typing bare 'exit' or 'quit' unconditionally quits.

We still ask for confirmation by default with C-d (EOF), since that can
happen accidentally quite easily (though users can always disble that safety
if desired).

Fix auto-quoting via ',' as the first character, which had broken in refactor.

Fix debugging with breakpoints.

URL: https://bugs.launchpad.net/ipython/+bug/381069

Fix bugs when readline is absent (pdb and quitting would crash badly)

https://bugs.launchpad.net/ipython/+bug/495173

Fix bug where output prompts were coming *after* output values.

This means that on exit/quit, we'll see something, so I added a simple 'bye'
to those. Not pretty, but easier than messing with the prompt logic.

Better handling of no-readline.

Fix broken multiline input - ugly fix, needs cleanup later.

See comment in code, my fix is nasty and should be cleaned up later, but
right now I can't do it (no time to understand the component tree machinery
well enough for a clean fix).

Minor fix to exit/quit

Update argparse to current SVN.

This fixes a few small bugs from upstream, but more importantly, syncs us so
we can add some local changes (see next commit)

Small improvements to argparse.

I've submitted these fixes upstream, let's hope they get included:

http://groups.google.com/group/argparse-devs/browse_thread/thread/2c03e4b3d28a3203

Cleaner access to log call.

At least now we're not walking the component tree manually... A bigger
refactor is still needed here, but this code is less brittle than multiple
.parent access.

Simpler/cleaner version of %who_ls.

Functionally equivalent but with simpler and faster code (I was fixing up
pylab support and saw this old code, fixed it up to more modern standards).

Testing '#:' attribute docstrings for sphinx.

This is supposed to work and give us class/instance docstrings, but it's not
yet. I have a question on the sphinx list about it, let's leave it in the
code until we figure out what we need to do to sphinx for it to work (it
doesn't hurt otherwise).

Added --gui to match %gui use, better docs and behavior for %pylab code.

Now both %gui and %pylab have identical syntax if used at startup as --gui
and --pylab. This can be useful to people setting up aliases or other
shell-based support.

Also cleaned up %pylab so it doesn't pollute the output of %who with all
loaded symbols.

Equivalent but simpler/faster code in loader.py.

Deactivate stray debug print statement.

Manage and propagate argv correctly.

All Application objects should take argv in their constructor, akin to how
the standard signature of C programs is "main(int argc, char *argv)". This
makes it possible to initialize them from code with different command-line
options (otherwise, they end up directly accessing sys.argv[1:] via
argparse).

Progress towards getting the test suite in shape again.

Work all over the place to get more tests to pass.

Close https://bugs.launchpad.net/ipython/+bug/503729

Removed stale tests that do not apply anymore.

Fix failing config test: https://bugs.launchpad.net/ipython/+bug/503731

Thanks to Brian for the fix!

Make testing easier by exposing a top-level test() function.

Fixes https://bugs.launchpad.net/ipython/+bug/504450

Work in multiple places to improve state of the test suite.

With these changes, on my system now all the test sub-suites pass except
for the Twisted one (see https://bugs.launchpad.net/ipython/+bug/504515 for
details on that one).

Improve test suite robustness by cleaning up stale processes when possible.

A few small fixes so ipythonx works, and PEP-8 cleanups I found along the way.

Minimal fixes so ipython-wx runs.

I've only done the most basic checks that it starts and simple things work,
this probably needs much more work in the future.

Cleaner error message in crash handler.

Move crash handling to the application level and simplify class structure.

Starting to try to take real advantage of the refactoring, to have generic
crash handling. This also lets us initialize the app without needing all
the self.attempt() wrappers, since now there's a good system-wide crash
handler at the app level (not inside the shell instance). I didn't yet
remove the attempt() method because we may have occasional uses for it (we
still do, but in one place only).

I also removed some extra class layers that weren't quite needed. Creating
classes solely for the purpose of passing parameters makes the code (IMO)
harder to understand, I kept getting lost in parts of the class hierarchy.
I think these changes provide the same flexibility but with easier to follow
code (less things to remember, basically). What I tried to do was to use
argument passing instead of inheritance for all cases I saw where the
inheritance wasn't really adding new functionality. In some cases, this
actually allowed me to remove methods that were effectively duplicated in
the subclasses.

Merging with upstream trunk.

Make IPython work if a recent numpy is not available.

I updated the numpy testing decorators module and didn't notice they
recently introduced dependencies that make it use the rest of numpy. We may
later want to revisit this, as my solution is ugly, but I simply hacked it
to load locally the code they added, which we now carry as
_numpy_testing_utils.

Ultimately the solution to this should be taken up with upstream numpy, so
that we all agree on a common set of testing decorators with no other
dependencies aside from the stdlib. But for now, this will let us move
forward as that discussion could take a while.

Make setupegg.py work on Windows.

Fixes: https://bugs.launchpad.net/ipython/+bug/505009

Fix deprecation warning from os.popen3.

Patch provided by Pim Schellart <p.schellart-AT-gmail.com>, slightly
modified to fix missing supbrocess references.

Fixes https://bugs.launchpad.net/ipython/+bug/488061

Fix broken %time magic.

Fixes https://bugs.launchpad.net/ipython/+bug/497187

Remove dead code accidentally left over in prior refactor.

Update version requirement to 2.5, since that's what we're using anyway.

Fixes https://bugs.launchpad.net/ipython/+bug/505090

Massive amount of work to improve the test suite, restores doctests.

After Brian's comments, I realized that our test machinery was NOT in
reality running all the ipython-syntax doctests we have. This is now fixed.
The test suite isn't completely passing, but this commit is for the
underlying machinery. I will now work on fixing as many broken tests as I
can.

Fixes https://bugs.launchpad.net/ipython/+bug/505071

Fix extensions test suite (small, but now it runs and passes!)

Fix test I broke when moving a file.

Fix config part of the test suite.

Fix failing doctest in utils.

Make all non-twisted tests pass.

Removed a test (builtins) that was already being done in test_run, fixed
broken doctest and fixed prefilter handling for empty buffers.

The only remaining failing test is now in twisted.

Disable completely two sets of Twisted tests so the suite runs.

The problem is that these tests are not just failing, they hang (even
ignoring Ctrl-C), so they prevent the suite from running at all. But I
don't know how to fix them, so I disabled them; this way others can at least
run the test suite, and hopefully Brian can pitch in with a fix.

Fix %history magics.

Fixes: https://bugs.launchpad.net/ipython/+bug/505343

Improve setuptools support.

Many thanks to Gael for the code/idea for the solution, see bug page for
details.

Fixes: https://bugs.launchpad.net/ipython/+bug/504968

Add file I forgot! Thanks to J. Hunter for report.

Patch by Beni Cherniavsky to improve tab-completion in emacs with 2.6.

https://bugs.launchpad.net/ipython/+bug/414967

Add transformers to understand code pasted with >>> or IPython prompts.

Now the following all work out of the box:

In [8]: In [6]: for i in range(5):
   ...: ...: print i,
   ...: ...:
   ...:
0 1 2 3 4

In [10]: >>> width = 20

In [11]: >>> height = 5*9

In [12]: >>> width * height
Out[12]: 900

And the history is still clean:

In [13]: %hist -n
[snipped]
for i in range(5):
    print i,

get_ipython().magic("hist -n")
width = 20
height = 5*9
width * height

This will be extremely useful when copy/pasting from interactive tutorials,
doctests and examples.

Also fixes %doctest_mode: https://bugs.launchpad.net/ipython/+bug/505404

Unify command-line usage information in one place.

Fixes: https://bugs.launchpad.net/ipython/+bug/505047

Simplify options handling code by using argparse argument_default.

The argparse.ArgumentParser class supports an instance-wide default, so we
can set that once instead of having every argument configured with the same
default always.

Apply argparse code simplification to all kernel scripts.

Update public revnum to make test installers for wider use.

At this point, all scripts, tests and doc build work on linux.

Update release tool to always produce zip file.

Fixes for python2.5 compatibility.

Minor fixes to build scripts.

build_rpm failing, don't know why; we may just stop providing rpms as those
are normally the distributor's problem.

Fix bug with autocall and multiline input recalled from readline buffer.

Reported by John Hunter on list.

Added new Tee class, that works much like Unix's 'tee' command.

I needed it to debug the IPython sphinx directive (next commit).

Also fixed some typos along the way.

Added ability to invoke pdb on IPython crashes.

For now it has to be turned on manually in the code, will add a flag later.
But very useful for debugging apps that use/extend IPython itself and can
possibly crash it.

Fix minor error that I saw in some odd cases, not sure how to test for it.

Unfortunately I don't have a reliable test case for this, it's deep in
traceback handling. But I did see a few times ultratb itself crash.

Ported the IPython Sphinx directive to 0.11.

This was originally written by John Hunter for the 0.10 API, now it works
with 0.11. We still need to automate its test suite, but at least now it
runs and the script itself can be executed as a test that produces screen
output and figures in a subdir.

Lots of work on exception handling, including tests for traceback printing.

We finally have some tests for various exception mode printing, via doctests
that exercise all three modes!

Also changed handling of sys.exit(X) to only print the summary message, as
SystemExit is most often a 'handled' exception. It can still be 100%
silenced via '%run -e', but now it's much less intrusive.

Added a new %tb magic to print the last available traceback with the current
xmode. One can then re-print the last traceback with more detail if
desired, without having to cause it again.

Changed %hist to default to NOT printing numbers, added -p and -o options.

-p: print python prompts before inputs
-o: print output as well

Together, these two make '%hist -op' an easy way to get doctest-valid
sessions generated from prior input.

Remove accidentally introduced runtime nose dependencies.

Fix bug with aliases, exposed by windows test suite.

Clean up and document better starting process of global IPython in tests.

Fix subtle bug in doctests (ipython namespace was being destroyed).

Exposed by windows test suite.

Various fixes for IPython.core tests to pass under win32.

Various fixes for test_genutils under win32, now all tests pass.

Remove executable bit, this is a module.

More fixes for win32 test suite.

Fix argument parsing bug in win32, clean up temp files in %hist doctest.

More win32 test fixes and a new test.

Mostly handling escaping of filenames across platforms.

Add new @onlyif decorator, the reverse of @skipif.

It's trivial, but it makes certain conditions much cleaner to express.

More win32 fixes and refinements to test suite.

Fix test suite when Twisted not available, cleanups to iptest for clarity.

Fix test failure when pexpect not available

Fix another win32 test failure.

As of this commit, the entire test suite passes on a system that only has:

- Python 2.6.4 from Python.org
- Pyreadline 1.5
- IPython (this branch)
- nose 0.11.1

Tests that require Twisted, graphics or other machinery get skipped, but
everything else runs and passes.

The glaring remaining problem on win32 is that right now, colored output is
broken for some reason (though prompts are OK). I'll try to fix that next.

Remove leftover print statement.

Add writeln convenience method to Term streams.

Fix exception color problems in win32.

In all the recent work to have the test suite play nice with doctest of full
ipython sessions, I inadvertedly started sending exceptions directly to
sys.stderr. On windows we MUST go via Term.cerr, which uses pyreadline to
handle color escapes, while sys.stderr just shows garbage on screen.

As of this revision, the test suite passes fully on win32 and linux, and
interactive use also seems OK on all fronts.

We're getting closer to RC status...

Move cleanup to main setup.py, where it belongs.

Distutils now generates .egg-info stuff even without setuptools, so we
should do the cleanup in the main script.

Fixes for test suite in win32 when all dependencies (esp. Twisted) are
installed.

Also activated testing.tools to be picked up by the test suite (was
excluded), this gives us a few more tests.

Status:

- On Linux, the full suite passes like before.

- On Win32, now that we have Twisted, we're seeing a few failures, because I
don't have the WinHPC server stuff. These should be easy for Brian to fix.
There are also two tests where the Skip nose exception isn't recognized by
Twisted, should also be easy. I'll file tickets for those.

Fix the two Twisted trial errors under win32.

I had incorrectly used a nose decorator instead of the twisted approach.

Changed trial skip to be a simple flag, now that I know this works.

I thought it needed to be a method, apparently not, use the simplest thing
that works.

Include 'kernel' in the nose tests as well; it picks up doctests that trial
doesn't.

Fix bug where python -c "import IPython; IPython.test()" could fail.

Depending on where one was located when typing it, it could fail. Made all
necessary paths absolute so this doesn't happen.

Added notes with ideas about new design for %magics.

Wrote up description with examples of our testing system.

Update revnum; ready for testing, will upload and send to the dev list.

Docs have been added, this is close to a reasonable RC for 0.11.

As of this revision, we're down to 0 errors on linux and 1 on windows (that
appears twice). The windows error is tracked here:

https://bugs.launchpad.net/ipython/+bug/506839

Clean up output of %who

Make an error message a bit easier to understand.

Fix %history to stop printing last line.

This was triggering a recursive calling of %hist in doctests, resulting in
spurious output from the test suite.

Closes https://bugs.launchpad.net/ipython/+bug/507705

Also minor pep-8 formatting cleanups.

Fix test I'd broken with change of reporting in %who.

Fix warning on startup if user didn't have personal copies of cluster config.

We use the builtin defaults in this case, we were just missing searching in
our own paths.

Also suppress deprecation warnings from Twisted.

Minimize stdout side effects in testing instances.

Fix frontend tests so they obtain the right global test IPython object.

Fix names so these doctests pass correctly when run in bare 'iptest IPython'

Let iptest pass arguments correctly to nose (in-process or in subprocess).

Fixes https://bugs.launchpad.net/ipython/+bug/507079

Also, remove relative imports from the iptest.py module so it can be run as
a script as well under 2.5 or 2.6.

Make it possible to run the tests from the source dir without installation.

Also clean up the handling of command-line options for iptest, so that all
options given at the top-level are correctly passed to nose.

Fixes: https://bugs.launchpad.net/ipython/+bug/507079

Moved twisted import into a function, so base exception classes in kernel
can be imported without twisted present.

Fixes to make test suite more robust on Fedora.

I fond some spurious warnings on Fedora, extra noise on stdout and other
small problems this commit fixes.

The test suite now runs cleanly on Fedora11 without Twisted available.

Do not call %run with quotes, since on Windows quotes aren't stripped.

On Posix, shlex strips quotes from filenames, but not on Windows. So if we
call %run "somefilename" on Win32, we get a 'file not found' error. This
problem was appearing for the test suite on win32.

Ensure that __builtin__ is always available and compute prompts from it.

__builtins__ (note the s) can change from module to dict somewhat at random,
see this for details:

http://mail.python.org/pipermail/python-dev/2001-April/014068.html

If this happened, our prompt computations could go out of whack in bizarre
ways, I saw it happen for continuation prompts after loading pylab, for
example. This change fixes it.

Comment out accidentally left over 'raise'.

This function promises NEVER to raise an exception, I was propagating it out
while debugging some odd behavior with __builtins__ and left it accidentally
in place.

Fix tests to return consistent results regardless of how they are called.

Now, calling:

- 'python iptest.py' from source dir or from src/IPython/testing should give
the same results

- 'python iptest.py IPython.SOMETHING' from either place should also give
the same results.

Make the test suite runnable without X11 connections.

This fix makes the test suite work in settings like a screen session that
can import but not initialize GTK.

Added %exit, %quit and %Quit as magics for consistency.

Doc updates to testing to reflect recent changes.

Updated revnumber to push an updated testing release out for this branch.

Include iptest.py in MANIFEST so it gets shipped.

Give good error message when starting tests if nose is missing.

Inform user at install time of minimal python requirements if not met.

Fixes: https://bugs.launchpad.net/ipython/+bug/505090

Robustness fixes in test suite machinery.

Added a module-level INSTALLED flag, which can be set to false if the test
suite is being run in-place (without ipython having been installed at all).
This is because how we call and import things must be done differently
depending on whether the code is installed or is being run in-place. The
only ones that can know this reliably are the entry-point scripts, so those
are responsible for setting this flag.

Also made the code that validates ipython in subprocesses report errors
better, by checking stderr for errors before validating stdout output, as
anything on stderr will be likely informative of the real problem.

Moved system info into its own utility, so we can use in in test suite too.

Added diagnostics printout at the end of the test suite.

This will make it easier for us to understand problem reports from users.