etcd+confd+nginx實現服務自動注冊與發現

一、etcd集群搭建

etcd集群搭建可參考：https://www.cnblogs.com/shenjianping/p/14399264.html

二、nginx的安裝與啟動

1、安裝前環境准備

yum install gcc-c++  #gcc編譯
yum install -y pcre pcre-devel  #nginx http正則
yum install -y zlib zlib-devel  #nginx http 包進行內容gzip
yum install -y openssl openssl-devel #nginx也支持https

2、下載源碼包

[root@localhost software]# wget -c https://nginx.org/download/nginx-1.12.0.tar.gz

3、解壓縮

[root@localhost software]# tar -xzvf nginx-1.12.0.tar.gz

4、配置安裝目錄

[root@localhost nginx-1.12.0]# ./configure --prefix=/usr/local/nginx

5、編譯和安裝

[root@localhost nginx-1.12.0]# make && make install

6、啟動nginx

[root@localhost nginx-1.12.0]# cd /usr/local/nginx/sbin/
[root@localhost sbin]# ./nginx 

此時可以通過ip地址進行訪問測試。

三、confd

1、下載與安裝

# 下載二進制文件
[root@localhost software]# wget https://github.com/kelseyhightower/confd/releases/download/v0.16.0/confd-0.16.0-linux-amd64

# 重命名並移動到PATH路徑下
[root@localhost software]# mv confd-0.16.0-linux-amd64 /usr/local/bin/confd

# 給予執行權限
[root@localhost software]# chmod +x /usr/local/bin/confd

# 驗證是否安裝成功
[root@localhost software]# confd --help

2、confd的配置

• 創建confdir

[root@localhost bin]# mkdir -p /etc/confd/{conf.d,templates}
[root@localhost bin]# ll /etc/confd/
total 0
drwxr-xr-x 2 root root 6 Feb 15 20:49 conf.d
drwxr-xr-x 2 root root 6 Feb 15 20:49 templates

創建conf.d和templates目錄：

• conf.d：confd的配置文件
• templates：配置模板Template

3、Template Resources

模板源TOML格式的配置文件，主要包含配置的生成邏輯，例如模板源，后端存儲對應的keys，命令執行等。默認目錄在/etc/confd/conf.d。

• /etc/confd/conf.d/myapp.toml

[template]
src = "nginx.tmpl"
dest = "/project/temp/mynginx.conf"
keys = [
  "/myapp/web/nginx/upstream",
  "/myapp/web/nginx/subdomain"
]
check_cmd = "/usr/local/nginx/sbin/nginx -t -c {{.src}}"
reload_cmd = "/usr/local/nginx/sbin/nginx reload"

4、Template

Template定義了單一應用配置的模板，默認存儲在 /etc/confd/templates目錄下。模板語法詳見：https://github.com/kelseyhightower/confd/blob/master/docs/templates.md

•  /etc/confd/templates/nginx.tmpl

pstream {{getv "/myapp/web/nginx/subdomain"}} {
{{range getvs "/myapp/web/nginx/upstream/*"}}
server {{.}};
{{end}}
}
server {
server_name {{getv "/myapp/web/nginx/subdomain"}}.example.com;
location / {
proxy_pass http://{{getv "/myapp/web/nginx/subdomain"}};
proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
}

四、進行測試

1、向etcd集群添加key和value

[root@localhost etcd-v3.3.10-linux-amd64]# export ETCDCTL_API=3

[root@localhost etcd-v3.3.10-linux-amd64]# ./etcdctl put /myapp/web/nginx/subdomain myapp

[root@localhost etcd-v3.3.10-linux-amd64]# ./etcdctl put /myapp/web/nginx/upstream/app1 "192.168.159.132:80"
[root@localhost etcd-v3.3.10-linux-amd64]# ./etcdctl put /myapp/web/nginx/upstream/app2 "192.168.159.133:80"

2、執行confd

[root@localhost etc]# confd -watch -backend etcdv3 -node http://192.168.159.128:2379
2021-02-16T13:02:26+08:00 localhost.localdomain confd[5674]: INFO Backend set to etcdv3
2021-02-16T13:02:26+08:00 localhost.localdomain confd[5674]: INFO Starting confd
...

3、查看結果

查看生成的配置文件/project/temp/mynginx.conf

upstream myapp {

server 192.168.159.132:80;
server 192.168.159.133:80;

}
server {
server_name myapp.example.com;
location / {
proxy_pass http://myapp;
proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}

4、整合nginx.conf

/project/temp/mynginx.conf是生成的配置，需要將其整合到nginx啟動項的配置中/usr/local/nginx/conf/nginx.conf，在http模塊下使用include命令即可：

http {
    include       mime.types;
    include /project/temp/mynginx.conf   #進行整合
    default_type  application/octet-stream;

    #log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
    #                  '$status $body_bytes_sent "$http_referer" '
    #                  '"$http_user_agent" "$http_x_forwarded_for"';

    #access_log  logs/access.log  main;

    sendfile        on;
    #tcp_nopush     on;
...
}

五、注意事項

• confd執行后台參數不正確

confd的執行confd -watch -backend etcdv3 -node http://192.168.159.128:2379中后台需要使用etcdv3，不能使用etcd，否則會報錯，其報錯內容如下：

[root@localhost etc]# confd -watch -backend etcd -node http://192.168.159.128:2379
2021-02-16T11:53:41+08:00 localhost.localdomain confd[5452]: INFO Backend set to etcd
2021-02-16T11:53:41+08:00 localhost.localdomain confd[5452]: INFO Starting confd
2021-02-16T11:53:41+08:00 localhost.localdomain confd[5452]: INFO Backend source(s) set to http://192.168.159.128:2379
2021-02-16T11:53:41+08:00 localhost.localdomain confd[5452]: ERROR 100: Key not found (/myapp) [61]

• confd執行域名地址不存在或者不正確

向etcd中添加nginx檢查無法通過的地址會報錯，所以一定要是配置正確的地址，其報錯內容如下：

[root@localhost etc]# confd -watch -backend etcdv3 -node http://192.168.159.128:2379
2021-02-16T11:55:12+08:00 localhost.localdomain confd[5458]: INFO Backend set to etcdv3
2021-02-16T11:55:12+08:00 localhost.localdomain confd[5458]: INFO Starting confd
2021-02-16T11:55:12+08:00 localhost.localdomain confd[5458]: INFO Backend source(s) set to http://192.168.159.128:2379
2021-02-16T11:55:12+08:00 localhost.localdomain confd[5458]: INFO Target config /project/temp/mynginx.conf out of sync
2021-02-16T11:55:12+08:00 localhost.localdomain confd[5458]: ERROR "/bin/sh: -c: line 0: syntax error near unexpected token `newline'\n/bin/sh: -c: line 0: `/usr/local/nginx/sbin/nginx -t -c <no value>'\n"
2021-02-16T11:55:12+08:00 localhost.localdomain confd[5458]: ERROR Config check failed: exit status 1