should interpret http 405 "not allowed" as "no smart server here" - breaks foreign branches on google code

http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.6
> 10.4.6 405 Method Not Allowed
>
> The method specified in the Request-Line is not allowed for the resource identified by the Request-URI. The response MUST include an Allow header containing a list of valid methods for the requested resource.

istm this would be reasonable to treat as TransportNotPossible, making sure that error includes the method and url. You might expect to get this on a post but in fact google seems to also serve it for a GET on an unexpected url.

ironically google code is now saying it's not available at this time, stymying my testing

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Martin Pool wrote:
> ironically google code is now saying it's not available at this time,
> stymying my testing
>

So the traceback seems to show a 503 rather than a 405, though perhaps
what you mean by this message.

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkso/YgACgkQJdeBCYSNAAMa0ACeJu+JpAIMjur8cxDA9J2jFMIf
KNoAnirpFphySMMfWSqXznN3TDnKKYmn
=08g8
-----END PGP SIGNATURE-----

2009/12/17 John A Meinel <email address hidden>:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Martin Pool wrote:
>> ironically google code is now saying it's not available at this time,
>> stymying my testing
>>
>
> So the traceback seems to show a 503 rather than a 405, though perhaps
> what you mean by this message.

Yes, that's correct. It seems to be persistently but intermittently
giving that too. I don't think we should have a special
interpretation for 503 because it really does mean "server down", but
the attached branch tries to give some data about why by showing the
error page text.

It's just possible this is some kind of banning imposed by google
because they dislike being probed, but because I can reproduce it from
an unrelated ip with wget I doubt it.

--
Martin <http://launchpad.net/~mbp/>

Something strange here: in my test today it gave just plain 404's looking for .bzr/branch-format. I'm sure I've seen other errors for that previously.

0.049 bzr arguments: [u'-Dhttp', u'-Derror', u'log', u'nosmart+http://spyderlib.googlecode.com/hg/']
0.111 looking for plugins in /home/mbp/.bazaar/plugins
...
0.284 * About to connect() to spyderlib.googlecode.com:80
0.470 > GET /hg/.bzr/branch-format
0.470 > Host: spyderlib.googlecode.com
> Accept: */*
> User-Agent: bzr/2.1.0dev5 (urllib)
> Connection: Keep-Alive
> Pragma: no-cache
> Cache-Control: max-age=0

1.017 < HTTP/1.1 404 Not Found
1.017 < Content-Type: text/html; charset=ISO-8859-1
< Date: Thu, 17 Dec 2009 05:26:42 GMT
< Server: hg_frontend
< Cache-Control: private, x-gzip-ok=""
< X-XSS-Protection: 0
< Transfer-Encoding: chunked

1.018 > GET /.bzr/branch-format
1.018 > Host: spyderlib.googlecode.com
> Accept: */*
> User-Agent: bzr/2.1.0dev5 (urllib)
> Connection: Keep-Alive
> Pragma: no-cache
> Cache-Control: max-age=0

1.295 < HTTP/1.1 404 Not Found
1.296 < Date: Thu, 17 Dec 2009 05:26:43 GMT
< Server: Apache
< Content-Type: text/html; charset=UTF-8
< Content-Length: 1372
< X-XSS-Protection: 0

With bzr+http it looks like this, showing my changes are not yet enough, and that Google is not having such a bad day as yesterday.

(I'm posting them here for later reference because of the intermittent failures.)

0.682 > POST /hg/.bzr/smart
0.682 > Content-Length: 6
> Connection: Keep-Alive
> Accept: */*
> User-Agent: bzr/2.1.0dev5 (urllib)
> Host: spyderlib.googlecode.com
> Pragma: no-cache
> Cache-Control: max-age=0
> Content-Type: application/x-www-form-urlencoded

0.986 < HTTP/1.1 405 Method Not Allowed
0.986 < Content-Type: text/html; charset=UTF-8
< Allow: GET, HEAD
< Date: Thu, 17 Dec 2009 05:29:16 GMT
< Server: hg_frontend
< Content-Length: 1404
< X-XSS-Protection: 0

0.988 hpss call: 'get', '.bzr/branch-format'
0.988 (to http://spyderlib.googlecode.com/hg/)
0.989 * About to connect() to spyderlib.googlecode.com:80
1.041 > POST /hg/.bzr/smart
1.041 > Content-Length: 94
> Connection: Keep-Alive
> Accept: */*
> User-Agent: bzr/2.1.0dev5 (urllib)
> Host: spyderlib.googlecode.com
> Pragma: no-cache
> Cache-Control: max-age=0
> Content-Type: application/x-www-form-urlencoded

1.346 < HTTP/1.1 405 Method Not Allowed
1.346 < Content-Type: text/html; charset=UTF-8
< Allow: GET, HEAD
< Date: Thu, 17 Dec 2009 05:29:16 GMT
< Server: hg_frontend
< Content-Length: 1404
< X-XSS-Protection: 0

1.424 Traceback (most recent call last):
  File "/home/mbp/bzr/497274-http-405/bzrlib/commands.py", line 826, in exception_to_return_code
    return the_callable(*args, **kwargs)
  File "/home/mbp/bzr/497274-http-405/bzrlib/commands.py", line 1027, in run_bzr
    ret = run(*run_argv)
  File "/home/mbp/bzr/497274-http-405/bzrlib/commands.py", line 638, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/home/mbp/bzr/497274-http-405/bzrlib/commands.py", line 1042, in ignore_pipe
    result = func(*args, **kwargs)
  File "/home/mbp/bzr/497274-http-405/bzrlib/builtins.py", line 2316, in run
    revision, file_list)
  File "/home/mbp/bzr/497274-http-405/bzrlib/log.py", line 1897, in _get_info_for_log_files
    tree, b, path = bzrdir.BzrDir.open_containing_tree_or_branch(file_list[0])
  File "/home/mbp/bzr/497274-http-405/bzrlib/bzrdir.py", line 971, in open_containing_tree_or_branch
    bzrdir, relpath = klass.open_containing(location)
  File "/home/mbp/bzr/497274-http-405/bzrlib/bzrdir.py", line 898, in open_containing
    return BzrDir.open_containing_from_transport(transport)
  File "/home/mbp/bzr/497274-http-405/bzrlib/bzrdir.py", line 919, in open_containing_from_transport
    result = BzrDir.open_from_transport(a_transport)
  File "/home/mbp/bzr/497274-http-405/bzrlib/bzrdir.py", line 873, in open_from_transport
    redirected)
  File "/home/mbp/bzr/497274-http-405/bzrlib/lazy_import.py", line 125, in __call__
    return obj(*args, **kwargs)
  File "/home/mbp/bzr/497274-http-405/bzrlib/transport/__init__.py", line 1644, in do_catching_redirections
    return action(transport)
  File "/home/mbp/bzr/497274-http-405/bzrlib/bzrdir.py", line 860, in find_format
    transport, _server_formats=_server_formats)
  File "/home/mbp/bzr/497274-http-405/bzrlib/bzrdir.py", line 1820, in find_format
    return format.probe_transpor...

Google seems to be changing this error code every now and again. Originally they returned a 401 when we tried to access .bzr/branch-format, even for public repositories.

>>>>> "Jelmer" == Jelmer Vernooij <email address hidden> writes:

    Jelmer> Google seems to be changing this error code every now and again.
    Jelmer> Originally they returned a 401 when we tried to access .bzr/branch-
    Jelmer> format, even for public repositories.

Haaa, thanks Jelmer, I couldn't remember exactly and I was
wondering...

My souvenir was that the real solution involved re-thinking the
way we probe for various VCS and I was a bit surprised that we
missed this easy fix at the time...

       Vincent

At the moment, as Vincent indicated in <https://code.edge.launchpad.net/~mbp/bzr/497274-http-405/+merge/16229>, this does seem to work in trunk when Google's not refusing everything, but it may depend on which http client implementation you're using.

Given that there doesn't seem to be a practical problem with Google anymore, I'm going to unassign and downgrade this.

Is there a fix for this? I'm getting exactly the same issue (on the same repository as well!), with bzr ver 2.2.1.

This still happens with bzr-hg against google.

The 405 also happens when trying to open a file on a remote http server (such as launchpad's librarian) - bzr will try to open that file with .bzr/branch-format appended to it.

This makes it impossible to "bzr merge <bundle-on-launcpad>".

It may be possible to pick up and reuse
<https://code.launchpad.net/~mbp/bzr/497274-http-405/+merge/16229>. I
abandoned that previously because I couldn't reproduce the problem
against code.google.com.

I am encountering this when I try to acccess github with HTTP:

$ bzr -Derror log https://github.com/mbrossard/threadpool.git
bzr: ERROR: bzrlib.errors.InvalidHttpResponse: Invalid http response for https://github.com/mbrossard/threadpool.git/info/refs: Unable to handle http code 405: Method Not Allowed

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/bzrlib/commands.py", line 946, in exception_to_return_code
    return the_callable(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/bzrlib/commands.py", line 1150, in run_bzr
    ret = run(*run_argv)
  File "/usr/local/lib/python2.7/site-packages/bzrlib/commands.py", line 699, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/usr/local/lib/python2.7/site-packages/bzrlib/commands.py", line 721, in run
    return self._operation.run_simple(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/bzrlib/cleanup.py", line 135, in run_simple
    self.cleanups, self.func, *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/bzrlib/cleanup.py", line 165, in _do_with_cleanups
    result = func(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/bzrlib/commands.py", line 1165, in ignore_pipe
    result = func(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/bzrlib/builtins.py", line 2458, in run
    revision, file_list, self.add_cleanup)
  File "/usr/local/lib/python2.7/site-packages/bzrlib/log.py", line 2096, in _get_info_for_log_files
    tree, b, path = bzrdir.BzrDir.open_containing_tree_or_branch(file_list[0])
  File "/usr/local/lib/python2.7/site-packages/bzrlib/bzrdir.py", line 931, in open_containing_tree_or_branch
    bzrdir, relpath = klass.open_containing(location)
  File "/usr/local/lib/python2.7/site-packages/bzrlib/bzrdir.py", line 874, in open_containing
    return BzrDir.open_containing_from_transport(transport)
  File "/usr/local/lib/python2.7/site-packages/bzrlib/bzrdir.py", line 895, in open_containing_from_transport
    result = BzrDir.open_from_transport(a_transport)
  File "/usr/local/lib/python2.7/site-packages/bzrlib/bzrdir.py", line 858, in open_from_transport
    redirected)
  File "/usr/local/lib/python2.7/site-packages/bzrlib/lazy_import.py", line 129, in __call__
    return obj(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/bzrlib/transport/__init__.py", line 1690, in do_catching_redirections
    return action(transport)
  File "/usr/local/lib/python2.7/site-packages/bzrlib/bzrdir.py", line 845, in find_format
    transport, _server_formats=_server_formats)
  File "/usr/local/lib/python2.7/site-packages/bzrlib/controldir.py", line 749, in find_format
    return prober.probe_transport(transport)
  File "/usr/local/lib/python2.7/site-packages/bzrlib/plugins/git/__init__.py", line 129, in probe_transport
    if not transport.has_any(['info/refs', '.git/HEAD', 'HEAD', 'objects', '.git/objects']):
  File "/usr/local/lib/python2.7/site-packages/bzrlib/transport/__init__.py", line 592, in has_any
    if self.has(relpath):
  File "/usr/local/lib/python2.7/site-packages/bzrlib/transport/http/_urllib.py", line 164, in has
    response = self._head(relpath)
  File...

Hi Janne,

You can work around this bug by using https+urllib rather than just https:, e.g.:

bzr log https+urllib://github.com/mbrossard/threadpool.git

Unfortunately "bzr log" can't ever work for Git repositories, because Git only allows fetching full packs, not just the revision details.

Hi Jelmer,

Thank you for your suggestion. Actually I was wanting to make a branch. I just used the "log" command in my example of the problem so that it matches the OP's bug report.

However, it appears that putting "+urllib" in the URL does not help in this case:

$ bzr branch https+urllib://github.com/mbrossard/threadpool.git
bzr: ERROR: Invalid http response for https://github.com/mbrossard/threadpool.git/info/refs: Unable to handle http code 405: Method Not Allowed
$

Maybe the suggested workaround works with the Google git but not with github? Or is it dependent on some specific versions?

On 14/09/11 00:46, Janne Snabb wrote:
> Hi Jelmer,
>
> Thank you for your suggestion. Actually I was wanting to make a branch.
> I just used the "log" command in my example of the problem so that it
> matches the OP's bug report.
>
> However, it appears that putting "+urllib" in the URL does not help in
> this case:
>
> $ bzr branch https+urllib://github.com/mbrossard/threadpool.git
> bzr: ERROR: Invalid http response for https://github.com/mbrossard/threadpool.git/info/refs: Unable to handle http code 405: Method Not Allowed
> $
>
> Maybe the suggested workaround works with the Google git but not with
> github? Or is it dependent on some specific versions?
Cloning git repositories over http requires the current trunk of dulwich
and bzr-git; are you running those?

With older versions of bzr-git you should still be able to clone that
repository, but by using git://github.com/mbrossard/threadpool.git

Cheers,

Jelmer

On Tue, 13 Sep 2011, Jelmer Vernooij wrote:

> Cloning git repositories over http requires the current trunk of dulwich
> and bzr-git; are you running those?

No. This is the problem then. I am using dulwich 0.8.0 and bzr-git
0.6.2. I believe these are the newest releases but not the trunk.

> With older versions of bzr-git you should still be able to clone that
> repository, but by using git://github.com/mbrossard/threadpool.git

Yeah, this works fine so far. Thank You!

--
Janne Snabb / EPIPE Communications
<email address hidden> - http://epipe.com/

the issue persists:
https://code.launchpad.net/~pythonxy/spyderlib/trunk

http://launchpadlibrarian.net/90780662/pythonxy-spyderlib-trunk.log

Hi Timmie,

The failure you're referring to is different from this bug.

So shall I report it elsewhere?

Am 23/01/12 21:28, schrieb Timmie:
> So shall I report it elsewhere?
>
Please file a bug against launchpad/bzr-hg if there isn't one already.

Cheers,

Jelmer

Hi all,
the automatic import from gogle code Mercurial works now:
https://code.launchpad.net/~pythonxy/spyderlib/trunk

Let's hope this persists.

A great thanks to Jelmer Vernooij (jelmer) for attending this for over a year now.

Hi Timmie,

Sorry to sound negative, but I don't think either of these issues has been resolved yet. The one you were hitting is a different bug than this one (it's a bzr-hg bug, not a bzr bug). No relevant changes have been made in bzr-hg trunk since you originally hit it so I suspect it's a bug that is related to incremental imports not working. You're very likely to hit it again when your upstream branch changes.