Etcd+Confd實現配置文件動態更新

簡介

如上圖是一個很簡單的架構，生產環境中經常會進行灰度發布，需要下掉一部分的節點。如果靠人工操作很容易錯誤，這里通過Etcd和Confd來實現nginx upstream的動態更新。

類似的，自動化部署時服務的環境變量等也可存入etcd(配置中心website頁面)，coredns等配置文件內容均可存入etcd，由confd動態刷新。

etcd: 分布式KV存儲系統，一般用於共享配置和服務注冊與發現。

confd：管理本地應用配置文件，使用etcd或consul存儲的數據渲染模板，還支持redis、zookeeper等, 通過watch定期監測對應的etcd中目錄變化，獲取最新的Value，然后渲染模板，更新配置文件。

安裝

• 安裝etcd

yum -y install etcdsystemctl start etcd

• 安裝confd

wget https://github.com/kelseyhightower/confd/releases/download/v0.16.0/confd-0.16.0-linux-arm64mkdir -p /etc/etcd/{conf.d,templates}mv confd-0.16.0-linux-arm64 /usr/bin/confd

• conf.d 目錄存放.toml配置文件
• templates 目錄存放.tmpl配置模版文件

配置

創建nginx配置和模版

配置文件cat conf.d/test.conf.toml

[
template]src = "test.conf.tmpl"
dest = "/tmp/test.conf"
keys = [ "/nginx",]
check_cmd = "/usr/sbin/nginx -t -c {{.src}}"
reload_cmd = "/usr/sbin/nginx -s reload"
]

模版文件cat templates/test.conf.tmpl

upstream www_{{getv "/nginx/www/server/server_name"}} {   
{{range getvs "/nginx/www/upstream/*"}}       
server {{.}};  
{{end}}
}
server {   
server_name       
{{getv "/nginx/www/server/server_name"}};  
location / {       
proxy_pass        http://www_{{getv "/nginx/www/server/server_name"}};        proxy_set_header Host $host;  
proxy_set_header X-Real-IP $remote_addr;  
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;   
proxy_set_header X-Forwarded-Proto https;     
proxy_redirect    off;  
}
}

配置etcd

etcdctl set /nginx/https/www/server/server_name test.com
etcdctl set /nginx/https/www/upstream/server1 192.168.1.110
etcdctl set /nginx/https/www/upstream/server2 192.168.1.111

啟動confd監聽

confd -watch -backend etcd -node http://127.0.0.1:2379

查看生產的nginx配置文件

 cat /tmp/test.confupstream www_test.com
 {        
 server 192.168.1.110;        
 server 192.168.1.111;}
 server {   
 server_name         test.com;    location / {        
 proxy_pass        http://www_test.com;        
 proxy_set_header Host $host;        
 proxy_set_header X-Real-IP $remote_addr;      
 proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;      
 proxy_set_header X-Forwarded-Proto https;        
 proxy_redirect    off;    }
 }

配置文件生成完成