Hi,

 

I am a newbie to the this list. After a lot of attempts to get this working wanted to try out  the mailing list for ideas. Please review this email in detail before suggesting to check password or other basic stuff.

 

My conlusion this far is that my environment on liberty is running glance v2. Nova is making calls to v2 of nova api at port 8794. This returns a error for image-list. Not sure if it is calling v1 of glance or not.

Glance v1 api is erroring out.

 

Looking for some pointers to below.

How do I add more debug to ensure nova is using glance api v2. I have seen some articles that state nova uses v1 api of glance, but not sure if that applies to liberty release.

If yes then how do I get v1 glance api working on the box.

 

See below for all the debug and issue details. Any help or response  is greatly appreciated.

nova image-list returns http 500 'glanceclient.exc.CommunicationError'.  Glance image-list works. Openstack image list works.  Nova service-list works.  cinder list works.

Glance v2 api works and is able to create images. Cinder is able to create volumes from the glance images.

All configuration files are enabled to debug and verbose . Nothing showing in the cinder or glance debug logs that would provide more details.

 

Glance is using version 2. IF I try version 1 api  glance will error out. Does glance api v1 works with liberty.  How do I enable this. My storage backend is ceph.

RBD is able to communicate to all volumes and all user ids are set correctly and have rrx for stores for glance, cinder and nova. Keyrings are present.

Not seeing any errors with  keystone .

 

Controller box has haproxy,mariadb, glance and cinder. nova and neutron are installed on a separate box.  Passwords in the mysql database  for glance,cinder,nova match what is in the config file.

Since I am using haproxy  for HA the end point ports are non default.  Services themselves are running on the default ports on each box.

For debug isolation purposes only using 1  controller (glance,cinder,keystone, haproxy) and 1 compute (nova and neutron)

 

 

# openstack image list

+--------------------------------------+--------+

| ID                                   | Name   |

+--------------------------------------+--------+

| f4040cc9-a2fa-45d5-9da3-52ffdde998a8 | cirros |

| de815da9-135a-4f8d-99ff-9e954672d775 | cirros |

 

# glance image-list

+--------------------------------------+--------+

| ID                                   | Name   |

+--------------------------------------+--------+

| de815da9-135a-4f8d-99ff-9e954672d775 | cirros |

| f4040cc9-a2fa-45d5-9da3-52ffdde998a8 | cirros |

+--------------------------------------+--------+

# nova service-list

+----+------------------+----------+----------+---------+-------+----------------------------+-----------------+

| Id | Binary           | Host     | Zone     | Status  | State | Updated_at                 | Disabled Reason |

+----+------------------+----------+----------+---------+-------+----------------------------+-----------------+

| 3  | nova-cert        | ostackn1 | internal | enabled | up    | 2016-01-11T19:29:52.000000 | -               |

| 5  | nova-consoleauth | ostackn1 | internal | enabled | up    | 2016-01-11T19:29:56.000000 | -               |

| 7  | nova-conductor   | ostackn1 | internal | enabled | up    | 2016-01-11T19:29:49.000000 | -               |

| 27 | nova-scheduler   | ostackn1 | internal | enabled | up    | 2016-01-11T19:29:51.000000 | -               |

| 29 | nova-compute     | ostackn1 | nova     | enabled | up    | 2016-01-11T19:29:54.000000 | -               |

| 31 | nova-network     | ostackn1 | internal | enabled | up    | 2016-01-11T19:29:49.000000 | -               |

+----+------------------+----------+----------+---------+-------+----------------------------+-----------------+

 

root@ostackn1:~# nova --debug image-list

DEBUG (session:198) REQ: curl -g -i -X GET http://10.1.4.54:35358/v2.0 -H "Accept: application/json" -H "User-Agent: python-keystoneclient"

INFO (connectionpool:205) Starting new HTTP connection (1): 10.1.4.54

DEBUG (connectionpool:385) "GET /v2.0 HTTP/1.1" 200 336

DEBUG (session:215) RESP: [200] Content-Length: 336 Vary: X-Auth-Token X-Distribution: Ubuntu Date: Mon, 11 Jan 2016 19:26:51 GMT Content-Type: application/json X-Openstack-Request-Id: req-e5ff4279-4632-4ae2-9f4f-af6cfdcb6727

RESP BODY: {"version": {"status": "stable", "updated": "2014-04-17T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v2.0+json"}], "id": "v2.0", "links": [{"href": "http://10.1.4.54:35358/v2.0/", "rel": "self"}, {"href": "http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}]}}

 

DEBUG (v2:86) Making authentication request to http://10.1.4.54:35358/v2.0/tokens

DEBUG (connectionpool:385) "POST /v2.0/tokens HTTP/1.1" 200 3066

DEBUG (session:198) REQ: curl -g -i -X GET http://10.1.4.54:8794/v2/ -H "User-Agent: python-novaclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}b62d2f18c957d5cebd47eb593f4a55f8ccec82a8"

INFO (connectionpool:205) Starting new HTTP connection (1): 10.1.4.54

DEBUG (connectionpool:385) "GET /v2/ HTTP/1.1" 200 374

DEBUG (session:215) RESP: [200] Date: Mon, 11 Jan 2016 19:26:53 GMT Content-Length: 374 Content-Type: application/json X-Compute-Request-Id: req-0a444962-ebec-4994-bd82-d00bacbfca60

RESP BODY: {"version": {"status": "SUPPORTED", "updated": "2011-01-21T11:33:21Z", "links": [{"href": "http://10.1.4.54:8794/v2/", "rel": "self"}, {"href": "http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}], "min_version": "", "version": "", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.compute+json;version=2"}], "id": "v2.0"}}

 

DEBUG (session:198) REQ: curl -g -i -X GET http://10.1.4.54:8794/v2/ab601ee07c3a4ed293dc04f5fa1d90a1/images/detail -H "User-Agent: python-novaclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}b62d2f18c957d5cebd47eb593f4a55f8ccec82a8"

DEBUG (connectionpool:385) "GET /v2/ab601ee07c3a4ed293dc04f5fa1d90a1/images/detail HTTP/1.1" 500 209

DEBUG (session:215) RESP: [500] Date: Mon, 11 Jan 2016 19:26:53 GMT Content-Length: 209 Content-Type: application/json; charset=UTF-8 X-Compute-Request-Id: req-ffab41dd-fb6b-40e0-b739-5705e7bd3b84

RESP BODY: {"computeFault": {"message": "Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.\n<class 'glanceclient.exc.CommunicationError'>", "code": 500}}

 

DEBUG (shell:905) Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.

<class 'glanceclient.exc.CommunicationError'> (HTTP 500) (Request-ID: req-ffab41dd-fb6b-40e0-b739-5705e7bd3b84)

Traceback (most recent call last):

  File "/usr/lib/python2.7/dist-packages/novaclient/shell.py", line 903, in main

    OpenStackComputeShell().main(argv)

  File "/usr/lib/python2.7/dist-packages/novaclient/shell.py", line 830, in main

    args.func(self.cs, args)

  File "/usr/lib/python2.7/dist-packages/novaclient/v2/shell.py", line 1115, in do_image_list

    image_list = cs.images.list(limit=limit)

  File "/usr/lib/python2.7/dist-packages/novaclient/v2/images.py", line 74, in list

    return self._list('/images%s%s' % (detail, query), 'images')

  File "/usr/lib/python2.7/dist-packages/novaclient/base.py", line 72, in _list

    _resp, body = self.api.client.get(url)

  File "/usr/lib/python2.7/dist-packages/keystoneclient/adapter.py", line 170, in get

    return self.request(url, 'GET', **kwargs)

  File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 93, in request

    raise exceptions.from_response(resp, body, url, method)

ClientException: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.

<class 'glanceclient.exc.CommunicationError'> (HTTP 500) (Request-ID: req-ffab41dd-fb6b-40e0-b739-5705e7bd3b84)

ERROR (ClientException): Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.

<class 'glanceclient.exc.CommunicationError'> (HTTP 500) (Request-ID: req-ffab41dd-fb6b-40e0-b739-5705e7bd3b84)

 

 

root@ostackc1:~# glance --os-image-api-version 2 image-list

+--------------------------------------+--------+

| ID                                   | Name   |

+--------------------------------------+--------+

| de815da9-135a-4f8d-99ff-9e954672d775 | cirros |

| f4040cc9-a2fa-45d5-9da3-52ffdde998a8 | cirros |

+--------------------------------------+--------+

root@ostackc1:~# glance --debug --os-image-api-version 1 image-list

curl -g -i -X GET -H 'Accept-Encoding: gzip, deflate' -H 'Accept: */*' -H 'User-Agent: python-glanceclient' -H 'Connection: keep-alive' -H 'X-Auth-Token: {SHA1}318d44da7749a1b808ca32024fedc494a0f76133' -H 'Content-Type: application/octet-stream' http://10.1.4.54:9293/v1/images/detail?sort_key=name&sort_dir=asc&limit=20

Request returned failure status 500.

Traceback (most recent call last):

  File "/usr/lib/python2.7/dist-packages/glanceclient/shell.py", line 700, in main

    args.func(client, args)

  File "/usr/lib/python2.7/dist-packages/glanceclient/v1/shell.py", line 123, in do_image_list

    utils.print_list(images, columns)

  File "/usr/lib/python2.7/dist-packages/glanceclient/common/utils.py", line 183, in print_list

    for o in objs:

  File "/usr/lib/python2.7/dist-packages/glanceclient/v1/images.py", line 254, in list

    for image in paginate(params, return_request_id):

  File "/usr/lib/python2.7/dist-packages/glanceclient/v1/images.py", line 238, in paginate

    images, resp = self._list(url, "images")

  File "/usr/lib/python2.7/dist-packages/glanceclient/v1/images.py", line 63, in _list

    resp, body = self.client.get(url)

  File "/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line 280, in get

    return self._request('GET', url, **kwargs)

  File "/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line 272, in _request

    resp, body_iter = self._handle_response(resp)

  File "/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line 93, in _handle_response

    raise exc.from_response(resp, resp.content)

HTTPInternalServerError: 500 Internal Server Error: The server has either erred or is incapable of performing the requested operation. (HTTP 500)

500 Internal Server Error: The server has either erred or is incapable of performing the requested operation. (HTTP 500)

 

keystone endpoint-list

 

+----------------------------------+-----------+---------------------------------------------+---------------------------------------------+----------------------------------------+----------------------------------+

|                id                |   region  |                  publicurl                  |                 internalurl                 |                adminurl                |            service_id            |

+----------------------------------+-----------+---------------------------------------------+---------------------------------------------+----------------------------------------+----------------------------------+

| 045de6640536445da86969241286fb01 | regionOne | http://10.1.4.54:8081/v1/AUTH_%(tenant_id)s | http://10.1.4.54:8081/v1/AUTH_%(tenant_id)s |         http://10.1.4.54:8081          | f3f2704480ea4b02bff61741e3ba734c |

| 0c399f7fd9f9456ea158679ac2d9e800 | regionOne |    http://10.1.4.54:8794/v2/%(tenant_id)s   |    http://10.1.4.54:8794/v2/%(tenant_id)s   | http://10.1.4.54:8794/v2/%(tenant_id)s | 6365cc8ba6c24c5fa624fc587e9d6614 |

| 1a8e073f90bf4450b0bb5fbc6d356b4a | regionOne |    http://10.1.4.54:8777/v1/%(tenant_id)s   |    http://10.1.4.54:8777/v1/%(tenant_id)s   | http://10.1.4.54:8777/v1/%(tenant_id)s | 8ff95f752a104f0f8f8e65ab02a550a2 |

| 33f8ec041a2c444d975317a9c74f5b7e | regionOne |            http://10.1.4.54:9293            |            http://10.1.4.54:9293            |         http://10.1.4.54:9293          | 622081c852b4498cb71b6b94651f8577 |

| e6887352979c41b7aad8a0a77d67d80b | regionOne |          http://10.1.4.54:5001/v2.0         |          http://10.1.4.54:5001/v2.0         |      http://10.1.4.54:35358/v2.0       | 2aeac7db1c154706ab2560d978073232 |

| ec164aed45c04499900098fa21ce9aac | regionOne |    http://10.1.4.54:8777/v2/%(tenant_id)s   |    http://10.1.4.54:8777/v2/%(tenant_id)s   | http://10.1.4.54:8777/v2/%(tenant_id)s | d857ec11a2414b95a5dc5b7f10cb7300 |

| f34cea0c5db04f17bde7b37e48d7372f | regionOne |            http://10.1.4.54:9697            |            http://10.1.4.54:9697            |         http://10.1.4.54:9697          | eee2db07934f4353b13211375a5df6a5 |

| ff54d6bd114f4e58bb1f26cf63f1d8af | regionOne |     http://10.1.4.54:8793/services/Cloud    |     http://10.1.4.54:8793/services/Cloud    |  http://10.1.4.54:8793/services/Cloud  | 2ad2865f6ef747a39112b2b58be13a1f |

+----------------------------------+-----------+---------------------------------------------+-------------------------------------------

 

Best Regards,

 

Rajiv Duggal

Office: 805.388.1711 x 528
Mobile: 805.312.3499
Email: rduggal@dexsystems.com| www.dex.com