Fetching lease fails with TTL

[hemna]

I'm using etc3dgw in openstack cinder and I'm running into an issue where the lease refresh is failing, due to a missing field.

stack@walt-stack-1  /opt/stack/logs/screen  pip freeze |grep etcd  1 ↵  482  12:23:14
etcd3==0.6.2
etcd3gw==0.1.0

---excerpt from the cinder log

2017-08-23 12:25:16.771 24793 ERROR root [-] Unexpected exception occurred 60 time(s)... retrying.: KeyError: 'TTL'
2017-08-23 12:25:16.771 24793 ERROR root Traceback (most recent call last):
2017-08-23 12:25:16.771 24793 ERROR root File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 250, in wrapper
2017-08-23 12:25:16.771 24793 ERROR root return infunc(*args, **kwargs)
2017-08-23 12:25:16.771 24793 ERROR root File "/usr/lib/python2.7/site-packages/tooz/coordination.py", line 187, in _beat_forever_until_stopped
2017-08-23 12:25:16.771 24793 ERROR root wait_until_next_beat = self._driver.heartbeat()
2017-08-23 12:25:16.771 24793 ERROR root File "/usr/lib/python2.7/site-packages/tooz/drivers/etcd3gw.py", line 191, in heartbeat
2017-08-23 12:25:16.771 24793 ERROR root lock.heartbeat()
2017-08-23 12:25:16.771 24793 ERROR root File "/usr/lib/python2.7/site-packages/tooz/drivers/etcd3gw.py", line 38, in wrapper
2017-08-23 12:25:16.771 24793 ERROR root return func(*args, **kwargs)
2017-08-23 12:25:16.771 24793 ERROR root File "/usr/lib/python2.7/site-packages/tooz/drivers/etcd3gw.py", line 154, in heartbeat
2017-08-23 12:25:16.771 24793 ERROR root self._lease.refresh()
2017-08-23 12:25:16.771 24793 ERROR root File "/usr/lib/python2.7/site-packages/etcd3gw/lease.py", line 64, in refresh
2017-08-23 12:25:16.771 24793 ERROR root return int(result['result']['TTL'])
2017-08-23 12:25:16.771 24793 ERROR root KeyError: 'TTL'
2017-08-23 12:25:16.771 24793 ERROR root

[dims]

@hemna in the

• openstack CI ? or local devstack ?
• what is the etcd version you are running? (not the python library version, but the etcd process itself)
• is this re-creatable? or random?

[hemna]

I'm running a single devstack deployment on 2 separate nodes to try and test out Cinder's active/active HA capability with the 3PAR cinder driver. I run into this while running some rally scenarios.

stack@walt-stack-2  /opt/stack/bin  ./etcd --version
etcd Version: 3.1.7
Git SHA: 43b7507
Go Version: go1.7.5
Go OS/Arch: linux/amd64

[dims]

@hemna As its going to be hard to recreate, could you please log the response we get from self.client.post? https://github.com/dims/etcd3-gateway/blob/master/etcd3gw/lease.py#L62

[jharbott]

Seems to happen in gate sporadically, see http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%20%5C%22KeyError%3A%20'TTL'%5C%22

From the timestamps it might be related to etcd getting slow:

http://logs.openstack.org/93/506093/4/check/gate-tempest-dsvm-cells-ubuntu-xenial/407192b/logs/screen-c-vol.txt.gz?#_Sep_22_00_24_50_163847

http://logs.openstack.org/93/506093/4/check/gate-tempest-dsvm-cells-ubuntu-xenial/407192b/logs/screen-etcd.txt.gz#_Sep_22_00_24_50_031662

[dims]

Looks like they have a FAQ just for this scenario : https://coreos.com/etcd/docs/latest/faq.html#performance