Hi,

I am a newbie to the this list. After a lot of attempts to get this working wanted to try out the mailing list for ideas. Please review this email in detail before suggesting to check password or other basic stuff.

My conlusion this far is that my environment on liberty is running glance v2. Nova is making calls to v2 of nova api at port 8794. This returns a error for image-list. Not sure if it is calling v1 of glance or not. Glance v1 api is erroring out.

Looking for some pointers to below. How do I add more debug to ensure nova is using glance api v2. I have seen some articles that state nova uses v1 api of glance, but not sure if that applies to liberty release. If yes then how do I get v1 glance api working on the box.

See below for all the debug and issue details. Any help or response is greatly appreciated. nova image-list returns http 500 'glanceclient.exc.CommunicationError'. Glance image-list works. Openstack image list works. Nova service-list works. cinder list works. Glance v2 api works and is able to create images. Cinder is able to create volumes from the glance images. All configuration files are enabled to debug and verbose . Nothing showing in the cinder or glance debug logs that would provide more details.

Glance is using version 2. IF I try version 1 api glance will error out. Does glance api v1 works with liberty. How do I enable this. My storage backend is ceph. RBD is able to communicate to all volumes and all user ids are set correctly and have rrx for stores for glance, cinder and nova. Keyrings are present. Not seeing any errors with keystone .

Controller box has haproxy,mariadb, glance and cinder. nova and neutron are installed on a separate box. Passwords in the mysql database for glance,cinder,nova match what is in the config file. Since I am using haproxy for HA the end point ports are non default. Services themselves are running on the default ports on each box. For debug isolation purposes only using 1 controller (glance,cinder,keystone, haproxy) and 1 compute (nova and neutron)

# openstack image list +--------------------------------------+--------+ | ID | Name | +--------------------------------------+--------+ | f4040cc9-a2fa-45d5-9da3-52ffdde998a8 | cirros | | de815da9-135a-4f8d-99ff-9e954672d775 | cirros |

# glance image-list +--------------------------------------+--------+ | ID | Name | +--------------------------------------+--------+ | de815da9-135a-4f8d-99ff-9e954672d775 | cirros | | f4040cc9-a2fa-45d5-9da3-52ffdde998a8 | cirros | +--------------------------------------+--------+ # nova service-list +----+------------------+----------+----------+---------+-------+----------------------------+-----------------+ | Id | Binary | Host | Zone | Status | State | Updated_at | Disabled Reason | +----+------------------+----------+----------+---------+-------+----------------------------+-----------------+ | 3 | nova-cert | ostackn1 | internal | enabled | up | 2016-01-11T19:29:52.000000 | - | | 5 | nova-consoleauth | ostackn1 | internal | enabled | up | 2016-01-11T19:29:56.000000 | - | | 7 | nova-conductor | ostackn1 | internal | enabled | up | 2016-01-11T19:29:49.000000 | - | | 27 | nova-scheduler | ostackn1 | internal | enabled | up | 2016-01-11T19:29:51.000000 | - | | 29 | nova-compute | ostackn1 | nova | enabled | up | 2016-01-11T19:29:54.000000 | - | | 31 | nova-network | ostackn1 | internal | enabled | up | 2016-01-11T19:29:49.000000 | - | +----+------------------+----------+----------+---------+-------+----------------------------+-----------------+

root@ostackn1:~# nova --debug image-list DEBUG (session:198) REQ: curl -g -i -X GET http://10.1.4.54:35358/v2.0 -H "Accept: application/json" -H "User-Agent: python-keystoneclient" INFO (connectionpool:205) Starting new HTTP connection (1): 10.1.4.54 DEBUG (connectionpool:385) "GET /v2.0 HTTP/1.1" 200 336 DEBUG (session:215) RESP: [200] Content-Length: 336 Vary: X-Auth-Token X-Distribution: Ubuntu Date: Mon, 11 Jan 2016 19:26:51 GMT Content-Type: application/json X-Openstack-Request-Id: req-e5ff4279-4632-4ae2-9f4f-af6cfdcb6727 RESP BODY: {"version": {"status": "stable", "updated": "2014-04-17T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v2.0+json"}], "id": "v2.0", "links": [{"href": "http://10.1.4.54:35358/v2.0/", "rel": "self"}, {"href": "http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}]}}

DEBUG (v2:86) Making authentication request to http://10.1.4.54:35358/v2.0/tokens DEBUG (connectionpool:385) "POST /v2.0/tokens HTTP/1.1" 200 3066 DEBUG (session:198) REQ: curl -g -i -X GET http://10.1.4.54:8794/v2/ -H "User-Agent: python-novaclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}b62d2f18c957d5cebd47eb593f4a55f8ccec82a8" INFO (connectionpool:205) Starting new HTTP connection (1): 10.1.4.54 DEBUG (connectionpool:385) "GET /v2/ HTTP/1.1" 200 374 DEBUG (session:215) RESP: [200] Date: Mon, 11 Jan 2016 19:26:53 GMT Content-Length: 374 Content-Type: application/json X-Compute-Request-Id: req-0a444962-ebec-4994-bd82-d00bacbfca60 RESP BODY: {"version": {"status": "SUPPORTED", "updated": "2011-01-21T11:33:21Z", "links": [{"href": "http://10.1.4.54:8794/v2/", "rel": "self"}, {"href": "http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}], "min_version": "", "version": "", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.compute+json;version=2"}], "id": "v2.0"}}

DEBUG (session:198) REQ: curl -g -i -X GET http://10.1.4.54:8794/v2/ab601ee07c3a4ed293dc04f5fa1d90a1/images/detail -H "User-Agent: python-novaclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}b62d2f18c957d5cebd47eb593f4a55f8ccec82a8" DEBUG (connectionpool:385) "GET /v2/ab601ee07c3a4ed293dc04f5fa1d90a1/images/detail HTTP/1.1" 500 209 DEBUG (session:215) RESP: [500] Date: Mon, 11 Jan 2016 19:26:53 GMT Content-Length: 209 Content-Type: application/json; charset=UTF-8 X-Compute-Request-Id: req-ffab41dd-fb6b-40e0-b739-5705e7bd3b84 RESP BODY: {"computeFault": {"message": "Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.\n<class 'glanceclient.exc.CommunicationError'>", "code": 500}}

DEBUG (shell:905) Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'glanceclient.exc.CommunicationError'> (HTTP 500) (Request-ID: req-ffab41dd-fb6b-40e0-b739-5705e7bd3b84) Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/novaclient/shell.py", line 903, in main OpenStackComputeShell().main(argv) File "/usr/lib/python2.7/dist-packages/novaclient/shell.py", line 830, in main args.func(self.cs, args) File "/usr/lib/python2.7/dist-packages/novaclient/v2/shell.py", line 1115, in do_image_list image_list = cs.images.list(limit=limit) File "/usr/lib/python2.7/dist-packages/novaclient/v2/images.py", line 74, in list return self._list('/images%s%s' % (detail, query), 'images') File "/usr/lib/python2.7/dist-packages/novaclient/base.py", line 72, in _list _resp, body = self.api.client.get(url) File "/usr/lib/python2.7/dist-packages/keystoneclient/adapter.py", line 170, in get return self.request(url, 'GET', **kwargs) File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 93, in request raise exceptions.from_response(resp, body, url, method) ClientException: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'glanceclient.exc.CommunicationError'> (HTTP 500) (Request-ID: req-ffab41dd-fb6b-40e0-b739-5705e7bd3b84) ERROR (ClientException): Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'glanceclient.exc.CommunicationError'> (HTTP 500) (Request-ID: req-ffab41dd-fb6b-40e0-b739-5705e7bd3b84)

root@ostackc1:~# glance --os-image-api-version 2 image-list +--------------------------------------+--------+ | ID | Name | +--------------------------------------+--------+ | de815da9-135a-4f8d-99ff-9e954672d775 | cirros | | f4040cc9-a2fa-45d5-9da3-52ffdde998a8 | cirros | +--------------------------------------+--------+ root@ostackc1:~# glance --debug --os-image-api-version 1 image-list curl -g -i -X GET -H 'Accept-Encoding: gzip, deflate' -H 'Accept: */*' -H 'User-Agent: python-glanceclient' -H 'Connection: keep-alive' -H 'X-Auth-Token: {SHA1}318d44da7749a1b808ca32024fedc494a0f76133' -H 'Content-Type: application/octet-stream' http://10.1.4.54:9293/v1/images/detail?sort_key=name&sort_dir=asc&li... Request returned failure status 500. Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/glanceclient/shell.py", line 700, in main args.func(client, args) File "/usr/lib/python2.7/dist-packages/glanceclient/v1/shell.py", line 123, in do_image_list utils.print_list(images, columns) File "/usr/lib/python2.7/dist-packages/glanceclient/common/utils.py", line 183, in print_list for o in objs: File "/usr/lib/python2.7/dist-packages/glanceclient/v1/images.py", line 254, in list for image in paginate(params, return_request_id): File "/usr/lib/python2.7/dist-packages/glanceclient/v1/images.py", line 238, in paginate images, resp = self._list(url, "images") File "/usr/lib/python2.7/dist-packages/glanceclient/v1/images.py", line 63, in _list resp, body = self.client.get(url) File "/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line 280, in get return self._request('GET', url, **kwargs) File "/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line 272, in _request resp, body_iter = self._handle_response(resp) File "/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line 93, in _handle_response raise exc.from_response(resp, resp.content) HTTPInternalServerError: 500 Internal Server Error: The server has either erred or is incapable of performing the requested operation. (HTTP 500) 500 Internal Server Error: The server has either erred or is incapable of performing the requested operation. (HTTP 500)

keystone endpoint-list

+----------------------------------+-----------+---------------------------------------------+---------------------------------------------+----------------------------------------+----------------------------------+ | id | region | publicurl | internalurl | adminurl | service_id | +----------------------------------+-----------+---------------------------------------------+---------------------------------------------+----------------------------------------+----------------------------------+ | 045de6640536445da86969241286fb01 | regionOne | http://10.1.4.54:8081/v1/AUTH_%(tenant_id)shttp://10.1.4.54:8081/v1/AUTH_%25(tenant_id)s | http://10.1.4.54:8081/v1/AUTH_%(tenant_id)shttp://10.1.4.54:8081/v1/AUTH_%25(tenant_id)s | http://10.1.4.54:8081 | f3f2704480ea4b02bff61741e3ba734c | | 0c399f7fd9f9456ea158679ac2d9e800 | regionOne | http://10.1.4.54:8794/v2/%(tenant_id)shttp://10.1.4.54:8794/v2/%25(tenant_id)s | http://10.1.4.54:8794/v2/%(tenant_id)shttp://10.1.4.54:8794/v2/%25(tenant_id)s | http://10.1.4.54:8794/v2/%(tenant_id)shttp://10.1.4.54:8794/v2/%25(tenant_id)s | 6365cc8ba6c24c5fa624fc587e9d6614 | | 1a8e073f90bf4450b0bb5fbc6d356b4a | regionOne | http://10.1.4.54:8777/v1/%(tenant_id)shttp://10.1.4.54:8777/v1/%25(tenant_id)s | http://10.1.4.54:8777/v1/%(tenant_id)shttp://10.1.4.54:8777/v1/%25(tenant_id)s | http://10.1.4.54:8777/v1/%(tenant_id)shttp://10.1.4.54:8777/v1/%25(tenant_id)s | 8ff95f752a104f0f8f8e65ab02a550a2 | | 33f8ec041a2c444d975317a9c74f5b7e | regionOne | http://10.1.4.54:9293 | http://10.1.4.54:9293 | http://10.1.4.54:9293 | 622081c852b4498cb71b6b94651f8577 | | e6887352979c41b7aad8a0a77d67d80b | regionOne | http://10.1.4.54:5001/v2.0 | http://10.1.4.54:5001/v2.0 | http://10.1.4.54:35358/v2.0 | 2aeac7db1c154706ab2560d978073232 | | ec164aed45c04499900098fa21ce9aac | regionOne | http://10.1.4.54:8777/v2/%(tenant_id)shttp://10.1.4.54:8777/v2/%25(tenant_id)s | http://10.1.4.54:8777/v2/%(tenant_id)shttp://10.1.4.54:8777/v2/%25(tenant_id)s | http://10.1.4.54:8777/v2/%(tenant_id)shttp://10.1.4.54:8777/v2/%25(tenant_id)s | d857ec11a2414b95a5dc5b7f10cb7300 | | f34cea0c5db04f17bde7b37e48d7372f | regionOne | http://10.1.4.54:9697 | http://10.1.4.54:9697 | http://10.1.4.54:9697 | eee2db07934f4353b13211375a5df6a5 | | ff54d6bd114f4e58bb1f26cf63f1d8af | regionOne | http://10.1.4.54:8793/services/Cloud | http://10.1.4.54:8793/services/Cloud | http://10.1.4.54:8793/services/Cloud | 2ad2865f6ef747a39112b2b58be13a1f | +----------------------------------+-----------+---------------------------------------------+-------------------------------------------

Best Regards,

Rajiv Duggal Office: 805.388.1711 x 528 Mobile: 805.312.3499 Email: rduggal@dexsystems.com|mailto:rduggal@dexsystems.com| www.dex.comhttp://www.dex.com