Confd watch etcdv3数据Token超时失效问题的分析

背景:

Confd的etcdv3 backend支持watch ETCD Server的数据变更,并试试应用模板刷新配置.

在配置https+username/password的时候,当etcd进程失败或者主备倒换后watch一直失败. server端持续提示"Invalid Token"

根因:

ETCD采用token方式做api鉴权. 

Confd支持JWT和Simple两种跟ETCD鉴权,JWT的安全新就不说了.

Simple依赖用户名和密码,这个鉴权只在首次连接的时候会触发.

backends/etcdv3/client.go:NewEtcdClient-> clientv3.New(cfg)

这个首次连接会自动认证并获取Token

vendor/github.com/coreos/etcd/clientv3/client.go: getToken

后面即使断链也不会再刷新,又由于Token存在有效期,在server端空闲时或者异常到倒换时delete token后将无法再次获取到新的数据.

规避方法:

在watch异常时调用getToken刷新一下.

步骤:

1\修改confd vender里etcd client代码对外提供RefreshToken接口(原来是内部接口,外面无法调用)

vendor/github.com/coreos/etcd/clientv3/client.go

func (c *Client) RefreshToken() error {
        return c.getToken(c.ctx)
}

2\watch异常时自动Refresh

2.1\不更新ETCD版本场景不区分异常码,直接更新(confd依赖的etcd版本过低,配套的api无法识别相关异常)

backends/etcdv3/client.go:createWatch

log.Warning("Watch to '%s' stopped at revision %d", prefix, w.revision)
// Disconnected or cancelled
// Wait for a moment to avoid reconnecting
// too quickly
time.Sleep(time.Duration(1) * time.Second)
//异常后直接刷新
refreshRst := client.RefreshToken()
log.Info("Token Refreshed. %v", refreshRst)
// Start from next revision so we are not missing anything
if w.revision > 0 {
        rch = client.Watch(context.Background(), prefix, clientv3.WithPrefix(),
                clientv3.WithRev(w.revision + 1))
} else {
        // Start from the latest revision
        rch = client.Watch(context.Background(), prefix, clientv3.WithPrefix(),
                                                clientv3.WithCreatedNotify())
}

2.2\ETCD更新到3.3.15以上(同步更新grpc配套版本),在出现PermissionDenied异常时刷新Token

if err := wresp.Err(); err != nil {
 log.Error("Watch error: %s", err.Error())
 if strings.Index(err.Error(),"code = PermissionDenied")>=0 {
  refreshRst := client.RefreshToken()
  log.Info("Token Refreshed. %v", refreshRst)
 }
}

后语:

期待ETCD内部能优化失效,链接重连时自动刷新.