Ubuntu
keystone package

After grizzly upgrade, EC2 API requests fail:Could not find: credential

Bug #1158563 reported by Adam Gandelman on 2013-03-21

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	keystone (Ubuntu)	Fix Released	Undecided	Unassigned

Bug Description

Upgrading a fairly standard Folsom setup to Grizzly using our packages. After the updated config files are put in place and database has been migrated, nova's EC2 API fails to authenticate requests with keystone. The OSAPI end point works just fine.

On the client:

$ euca-describe-instances
Unauthorized: Failure communicating with keystone

On the keystone server, with debug enabled:

2013-03-21 18:53:36 2013-03-21 18:53:36 2013-03-21 18:53:36 2013-03-21 18:53:36 2013-03-21 18:53:36 2013-03-21 18:53:36 2013-03-21 18:53:36 2013-03-21 18:53:36 2013-03-21 18:53:36 2013-03-21 18:53:36 2013-03-21 18:53:36 2013-03-21 18:53:36 2013-03-21 18:53:36 2013-03-21 18:53:36 2013-03-21 18:53:36 2013-03-21 18:53:36 2013-03-21 18:53:36 2013-03-21 18:53:36 2013-03-21 18:53:36 2013-03-21 18:53:36 2013-03-21 18:53:36 2013-03-21 18:53:36 2013-03-21 18:53:36 2013-03-21 18:53:36 2013-03-21 18:53:36 2013-03-21 18:53:36 2013-03-21 18:53:36 2013-03-21 18:53:36 2013-03-21 18:53:36 2013-03-21 18:53:36 2013-03-21 18:53:36 2013-03-21 18:53:36 2013-03-21 18:53:36 2013-03-21 18:53:36 2013-03-21 18:53:36 2013-03-21 18:53:36 2013-03-21 18:53:36 2013-03-21 18:53:36 2013-03-21 18:53:36 2013-03-21 18:53:36 2013-03-21 18:53:36 2013-03-21 18:53:36 2013-03-21 18:53:36 DEBUG [keystone.common.wsgi] ******************** REQUEST ENVIRON ********************
DEBUG [keystone.common.wsgi] SCRIPT_NAME = /v2.0
DEBUG [keystone.common.wsgi] webob.adhoc_attrs = {'response': <Response at 0x2d64250 200 OK>}
DEBUG [keystone.common.wsgi] REQUEST_METHOD = POST
DEBUG [keystone.common.wsgi] PATH_INFO = /ec2tokens
DEBUG [keystone.common.wsgi] SERVER_PROTOCOL = HTTP/1.0
DEBUG [keystone.common.wsgi] CONTENT_LENGTH = 436
DEBUG [keystone.common.wsgi] eventlet.posthooks = []
DEBUG [keystone.common.wsgi] RAW_PATH_INFO = /v2.0/ec2tokens
DEBUG [keystone.common.wsgi] REMOTE_ADDR = 192.168.20.50
DEBUG [keystone.common.wsgi] eventlet.input = <eventlet.wsgi.Input object at 0x2d56050>
DEBUG [keystone.common.wsgi] wsgi.url_scheme = http
DEBUG [keystone.common.wsgi] webob._body_file = (<LimitedLengthFile(<eventlet.wsgi.Input object at 0x2d56050>, maxlen=436)>, <eventlet.wsgi.Input object at 0x2d56050>)
DEBUG [keystone.common.wsgi] SERVER_PORT = 5000
DEBUG [keystone.common.wsgi] wsgi.input = <_io.BytesIO object at 0x2d5c1d0>
DEBUG [keystone.common.wsgi] openstack.context = {'token_id': None, 'is_admin': False}
DEBUG [keystone.common.wsgi] HTTP_HOST = test-09.os.magners.qa.lexington:5000
DEBUG [keystone.common.wsgi] wsgi.multithread = True
DEBUG [keystone.common.wsgi] openstack.params = {u'ec2Credentials': {u'access': u'8e283c4e394247fbbabe5474cdbda9e4', u'host': u'test-02.os.magners.qa.lexington:8773', u'verb': u'POST', u'params': {u'SignatureVersion': u'2', u'AWSAccessKeyId': u'8e283c4e394247fbbabe5474cdbda9e4', u'Timestamp': u'2013-03-21T22:53:36Z', u'SignatureMethod': u'HmacSHA256', u'Version': u'2010-08-31', u'Action': u'DescribeInstances'}, u'signature': u'gEJ42tacqfUrqvgpj2ouqg3T+aAiwzu6c5LWMrRQDEA=', u'path': u'/services/Cloud/'}}
DEBUG [keystone.common.wsgi] wsgi.version = (1, 0)
DEBUG [keystone.common.wsgi] SERVER_NAME = 192.168.20.57
DEBUG [keystone.common.wsgi] GATEWAY_INTERFACE = CGI/1.1
DEBUG [keystone.common.wsgi] wsgi.run_once = False
DEBUG [keystone.common.wsgi] wsgi.errors = <open file '<stderr>', mode 'w' at 0x7f0bbbf08270>
DEBUG [keystone.common.wsgi] wsgi.multiprocess = False
DEBUG [keystone.common.wsgi] webob.is_body_seekable = True
DEBUG [keystone.common.wsgi] CONTENT_TYPE = application/json
DEBUG [keystone.common.wsgi] HTTP_ACCEPT_ENCODING = identity
DEBUG [keystone.common.wsgi]
DEBUG [keystone.common.wsgi] ******************** REQUEST BODY ********************
DEBUG [keystone.common.wsgi] {"ec2Credentials": {"access": "8e283c4e394247fbbabe5474cdbda9e4", "host": "test-02.os.magners.qa.lexington:8773", "verb": "POST", "params": {"SignatureVersion": "2", "AWSAccessKeyId": "8e283c4e394247fbbabe5474cdbda9e4", "Timestamp": "2013-03-21T22:53:36Z", "SignatureMethod": "HmacSHA256", "Version": "2010-08-31", "Action": "DescribeInstances"}, "signature": "gEJ42tacqfUrqvgpj2ouqg3T+aAiwzu6c5LWMrRQDEA=", "path": "/services/Cloud/"}}
DEBUG [keystone.common.wsgi]
DEBUG [keystone.common.wsgi] arg_dict: {}
WARNING [keystone.common.wsgi] Could not find: credential-8e283c4e394247fbbabe5474cdbda9e4
DEBUG [keystone.common.wsgi] ******************** RESPONSE HEADERS ********************
DEBUG [keystone.common.wsgi] Vary = X-Auth-Token
DEBUG [keystone.common.wsgi] Content-Type = application/json
DEBUG [keystone.common.wsgi] Content-Length = 120
DEBUG [keystone.common.wsgi]
DEBUG [keystone.common.wsgi] ******************** RESPONSE BODY ********************
DEBUG [keystone.common.wsgi] {"error": {"message": "Could not find: credential-8e283c4e394247fbbabe5474cdbda9e4", "code": 404, "title": "Not Found"}}
INFO [access] 192.168.20.50 - - [21/Mar/2013:22:53:36 +0000] "POST http://test-09.os.magners.qa.lexington:5000/v2.0/ec2tokens HTTP/1.0" 404 120
DEBUG [eventlet.wsgi.server] 192.168.20.50 - - [21/Mar/2013 18:53:36] "POST /v2.0/ec2tokens HTTP/1.1" 404 256 0.010670

The nova-api-ec2.log:

2013-03-21 18:54:28.516 13600 ERROR nova.api.ec2 [-] Unauthorized: Failure communicating with keystone
2013-03-21 18:54:28.516 13600 INFO nova.api.ec2 [-] 0.14199s 192.168.20.1 POST /services/Cloud/ None:None 400 [Boto/2.3.0 (linux2)] application/x-www-form-urlencoded text/xml
2013-03-21 18:54:28.517 13600 INFO nova.ec2.wsgi.server [-] 192.168.20.1 "POST /services/Cloud/ HTTP/1.1" status: 400 len: 327 time: 0.0149109

Tags:

Related branches

lp:~gandelman-a/ubuntu/raring/keystone/lp1158563

Merged into lp:~openstack-ubuntu-testing/keystone/grizzly at revision 191

Openstack Ubuntu Testers: Pending requested 2013-04-04

lp:~openstack-ubuntu-testing/keystone/grizzly

lp:ubuntu/raring-proposed/keystone

Adam Gandelman (gandelman-a) on 2013-03-21

tags:

added: upgrade

Michael Still (mikal) on 2013-04-02

tags:

added: ec2

Revision history for this message

Adam Gandelman (gandelman-a) wrote on 2013-04-04:

This is total configuration / packaging error. After the package upgrade, the EC2 was set to use the KVS backend, when it was originally using SQL.

Adam Gandelman (gandelman-a) on 2013-04-04

Changed in keystone (Ubuntu):
status:	New → Confirmed

Revision history for this message

Michael Still (mikal) wrote on 2013-04-04:

@Adam -- can we therefore remove the upstream tasks from this bug?

Revision history for this message

Adam Gandelman (gandelman-a) wrote on 2013-04-04:

Michael, I'd love nothing more, but LP seems to be timing out anytime I try touching those tasks. :\

no longer affects:	keystone
no longer affects:	nova
no longer affects:	nova (Ubuntu)

Revision history for this message

Launchpad Janitor (janitor) wrote on 2013-04-06:

This bug was fixed in the package keystone - 1:2013.1-0ubuntu1

---------------
keystone (1:2013.1-0ubuntu1) raring; urgency=low

  [ Adam Gandelman ]
  * debian/patches/sql_connection.patch: Ensure SQL by default for all
    backends. (LP: #1158563)
  * debian/rules: Reinstate use of test_overrides.conf to target upstream
    defaults when running unit tests.

[ Chuck Short ]
* New upstream release.
-- Chuck Short <email address hidden> Fri, 05 Apr 2013 22:32:17 -0500

Changed in keystone (Ubuntu):
status:	Confirmed → Fix Released

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.

Ubuntukeystone package

After grizzly upgrade, EC2 API requests fail:Could not find: credential

Bug Description

Related branches

Other bug subscribers

Remote bug watches

Ubuntu
keystone package