使用 etcdadm 部署 etcd 集群

发布时间： 更新时间： 总字数：1552 阅读时间：4m 作者： IP上海 分享 网址

etcdadm是一个命令行的工具，用于操作，集群。它可以很容易地创建一个新的集群，将一个成员添加或删除某成员从现有的集群。它的用户经验的启发通过 kubeadm

安装

  • 要安装 etcd 的环境参考
  • 下载 etcd
# 下载页面 https://github.com/kubernetes-sigs/etcdadm/releases
wget https://github.com/kubernetes-sigs/etcdadm/releases/download/v0.1.5/etcdadm-linux-amd64
mv etcdadm-linux-amd64 etcdadm
chmod a+x etcdadm
  • /etc/hosts 配置
172.17.0.3      etcd1
172.17.0.4      etcd2
172.17.0.5      etcd3
  • 复制 etcd 到所有节点
for ((i=1; i<=3; i++)); do
  docker cp etcdadm etcd${i}:/usr/local/bin/
done

help

etcdadm --help ... 
$ etcdadm --help
Tool to bootstrap etcdadm on the host

Usage:
  etcdadm [command]

Available Commands:
  download    Download etcd binary
  help        Help about any command
  info        Information about the local etcd member
  init        Initialize a new etcd cluster
  join        Join an existing etcd cluster
  reset       Remove this etcd member from the cluster and uninstall etcd
  version     Print version information

Flags:
  -h, --help               help for etcdadm
  -l, --log-level string   set log level for output, permitted values debug, info, warn, error, fatal and panic (default "info")

Use "etcdadm [command] --help" for more information about a command.

初始化集群

etcdadm help init ... 
$ etcdadm help init
Initialize a new etcd cluster

Usage:
  etcdadm init [flags]

Flags:
      --certs-dir string                    certificates directory (default "/etc/etcd/pki")
      --disk-priorities stringArray         Setting etcd disk priority (default [Nice=-10,IOSchedulingClass=best-effort,IOSchedulingPriority=2])
      --download-connect-timeout duration   Maximum time in seconds that you allow the connection to the server to take. (default 10s)
  -h, --help                                help for init
      --install-dir string                  install directory (default "/opt/bin/")
      --name string                         etcd member name
      --release-url string                  URL used to download etcd (default "https://github.com/coreos/etcd/releases/download")
      --server-cert-extra-sans strings      optional extra Subject Alternative Names for the etcd server signing cert, can be multiple comma separated DNS names or IPs
      --skip-hash-check                     Ignore snapshot integrity hash value (required if copied from data directory)
      --snapshot string                     Etcd v3 snapshot file used to initialize member
      --version string                      etcd version (default "3.4.9")

Global Flags:
  -l, --log-level string   set log level for output, permitted values debug, info, warn, error, fatal and panic (default "info")

说明：

  • etcd 证书地址 /etc/etcd/pki
  • 配置磁盘优先级 --disk-priorities
  • 默认安装目录 --install-dir /opt/bin/
  • 默认 etcd 版本 3.4.9，使用 --version 3.4.9 指定版本

在 etcd 1 操作

# 使用 https://gh-proxy.com/ 加速
$ etcdadm init --release-url https://gh-proxy.com/https://github.com/coreos/etcd/releases/download --server-cert-extra-sans "etcd1,etcd2,etcd3,172.17.0.3,172.17.0.4,172.17.0.5" --version 3.5.13
  • 详细日志如下
etcdadm init log ... 
root@etcd1:~# etcdadm init --release-url https://gh-proxy.com/https://github.com/coreos/etcd/releases/download --server-cert-extra-sans "etcd1,etcd2,etcd3,172.17.0.3,172.17.0.4,172.17.0.5" --version 3.5.13
INFO[0000] [install] Artifact not found in cache. Trying to fetch from upstream: https://gh-proxy.com/https://github.com/coreos/etcd/releases/download
INFO[0000] [install] Downloading & installing etcd https://gh-proxy.com/https://github.com/coreos/etcd/releases/download from 3.5.13 to /var/cache/etcdadm/etcd/v3.5.13
INFO[0000] [install] downloading etcd from https://gh-proxy.com/https://github.com/coreos/etcd/releases/download/v3.5.13/etcd-v3.5.13-linux-amd64.tar.gz to /var/cache/etcdadm/etcd/v3.5.13/etcd-v3.5.13-linux-amd64.tar.gz
######################################################################## 100.0%
INFO[0000] [install] extracting etcd archive /var/cache/etcdadm/etcd/v3.5.13/etcd-v3.5.13-linux-amd64.tar.gz to /tmp/etcd713259203
INFO[0000] [install] verifying etcd 3.5.13 is installed in /opt/bin/
INFO[0000] [certificates] creating PKI assets
INFO[0000] creating a self signed etcd CA certificate and key files
[certificates] Generated ca certificate and key.
INFO[0000] creating a new server certificate and key files for etcd
[certificates] Generated server certificate and key.
[certificates] server serving cert is signed for DNS names [etcd1 etcd2 etcd3 etcd1] and IPs [172.17.0.3 172.17.0.4 172.17.0.5 172.17.0.3 127.0.0.1]
INFO[0000] creating a new certificate and key files for etcd peering
[certificates] Generated peer certificate and key.
[certificates] peer serving cert is signed for DNS names [etcd1] and IPs [172.17.0.3]
INFO[0001] creating a new client certificate for the etcdctl
[certificates] Generated etcdctl-etcd-client certificate and key.
INFO[0001] creating a new client certificate for the apiserver calling etcd
[certificates] Generated apiserver-etcd-client certificate and key.
[certificates] valid certificates and keys now exist in "/etc/etcd/pki"
INFO[0002] [health] Checking local etcd endpoint health
INFO[0002] [health] Local etcd endpoint is healthy
INFO[0002] To add another member to the cluster, copy the CA cert/key to its certificate dir and run:
INFO[0002] 	etcdadm join https://172.17.0.3:2379
  • 当前节点信息
etcdadm info
{
  "ID": 6406391081376168596,
  "name": "etcd1",
  "peerURLs": [
    "https://172.17.0.3:2380"
  ],
  "clientURLs": [
    "https://172.17.0.3:2379"
  ]
}
  • 配置 /etc/etcd/
etcd configure ... 
root@etcd1:~# cat /etc/etcd/
etcd.env     etcdctl.env  pki/
root@etcd1:~# cat /etc/etcd/etcd.env
ETCD_NAME=etcd1

# Initial cluster configuration
ETCD_INITIAL_CLUSTER=etcd1=https://172.17.0.3:2380
ETCD_INITIAL_CLUSTER_TOKEN=45f0428a
ETCD_INITIAL_CLUSTER_STATE=new

# Peer configuration
ETCD_INITIAL_ADVERTISE_PEER_URLS=https://172.17.0.3:2380
ETCD_LISTEN_PEER_URLS=https://172.17.0.3:2380

ETCD_CLIENT_CERT_AUTH=true
ETCD_PEER_CERT_FILE=/etc/etcd/pki/peer.crt
ETCD_PEER_KEY_FILE=/etc/etcd/pki/peer.key
ETCD_PEER_TRUSTED_CA_FILE=/etc/etcd/pki/ca.crt

# Client/server configuration
ETCD_ADVERTISE_CLIENT_URLS=https://172.17.0.3:2379
ETCD_LISTEN_CLIENT_URLS=https://172.17.0.3:2379,https://127.0.0.1:2379

ETCD_PEER_CLIENT_CERT_AUTH=true
ETCD_CERT_FILE=/etc/etcd/pki/server.crt
ETCD_KEY_FILE=/etc/etcd/pki/server.key
ETCD_TRUSTED_CA_FILE=/etc/etcd/pki/ca.crt

# Other
ETCD_DATA_DIR=/var/lib/etcd
ETCD_STRICT_RECONFIG_CHECK=true
GOMAXPROCS=6

root@etcd1:~# cat /etc/etcd/etcdctl.env
export ETCDCTL_API=3

export ETCDCTL_CACERT=/etc/etcd/pki/ca.crt
export ETCDCTL_CERT=/etc/etcd/pki/etcdctl-etcd-client.crt
export ETCDCTL_KEY=/etc/etcd/pki/etcdctl-etcd-client.key

export ETCDCTL_DIAL_TIMEOUT=3s
  • 自动生成的证书
$ ls -lhart /etc/etcd/pki
total 48K
ca.key
ca.crt
server.key
server.crt
peer.key
peer.crt
etcdctl-etcd-client.key
etcdctl-etcd-client.crt
apiserver-etcd-client.key
apiserver-etcd-client.crt
  • 查看日志
journalctl -u etcd -f
  • 启动文件
$ systemctl cat etcd.service
# /etc/systemd/system/etcd.service
[Unit]
Description=etcd
Documentation=https://github.com/coreos/etcd
Conflicts=etcd-member.service
Conflicts=etcd2.service

[Service]
EnvironmentFile=/etc/etcd/etcd.env
ExecStart=/opt/bin/etcd

Type=notify
TimeoutStartSec=0
Restart=on-failure
RestartSec=5s

LimitNOFILE=65536
Nice=-10
IOSchedulingClass=best-effort
IOSchedulingPriority=2
MemoryLow=200M

[Install]
WantedBy=multi-user.target
  • etcdctl 客户端
ln -s /opt/bin/etcdctl /usr/local/bin/

添加节点

  • 初始化目录
mkdir -p /etc/etcd/pki/ /var/cache/etcdadm/etcd/v3.5.13/
  • 从安装过的节点分发 CA 证书
rsync -avR /etc/etcd/pki/ca.* <Member IP address>:/
# rsync -avR /etc/etcd/pki/ca.* 172.17.0.4:/
  • 复制缓存的安装包，提前缓存帮助 etcdadm help download
rsync -avR /var/cache/etcdadm/etcd/v3.5.13/etcd-v3.5.13-linux-amd64.tar.gz <Member IP address>:/
# rsync -avR /var/cache/etcdadm/etcd/v3.5.13/etcd-v3.5.13-linux-amd64.tar.gz 172.17.0.4:/
  • 新节点加入集群
etcdadm help join ... 
$ etcdadm help join
Join an existing etcd cluster

Usage:
  etcdadm join [flags]

Flags:
      --certs-dir string                 certificates directory (default "/etc/etcd/pki")
      --disk-priorities stringArray      Setting etcd disk priority (default [Nice=-10,IOSchedulingClass=best-effort,IOSchedulingPriority=2])
  -h, --help                             help for join
      --install-dir string               install directory (default "/opt/bin/")
      --name string                      etcd member name
      --release-url string               URL used to download etcd (default "https://github.com/coreos/etcd/releases/download")
      --retry                            Enable or disable backoff retry when join etcd member to cluster (default true)
      --server-cert-extra-sans strings   optional extra Subject Alternative Names for the etcd server signing cert, can be multiple comma separated DNS names or IPs
      --version string                   etcd version (default "3.4.9")

Global Flags:
  -l, --log-level string   set log level for output, permitted values debug, info, warn, error, fatal and panic (default "info")

etcdadm join <endpoint> --release-url https://gh-proxy.com/https://github.com/coreos/etcd/releases/download
etcdadm join log ... 
root@etcd2:~# etcdadm join https://172.17.0.3:2379 --release-url https://gh-proxy.com/https://github.com/coreos/etcd/releases/download --server-cert-extra-sans "etcd1,etcd2,etcd3,172.17.0.3,172.17.0.4,172.17.0.5"  --log-level debug --version 3.5.13
INFO[0000] [certificates] creating PKI assets
INFO[0000] creating a self signed etcd CA certificate and key files
[certificates] Using the existing ca certificate and key.
INFO[0000] creating a new server certificate and key files for etcd
[certificates] Using the existing server certificate and key.
INFO[0000] creating a new certificate and key files for etcd peering
[certificates] Using the existing peer certificate and key.
INFO[0000] creating a new client certificate for the etcdctl
[certificates] Using the existing etcdctl-etcd-client certificate and key.
INFO[0000] creating a new client certificate for the apiserver calling etcd
[certificates] Using the existing apiserver-etcd-client certificate and key.
[certificates] valid certificates and keys now exist in "/etc/etcd/pki"
INFO[0000] [membership] Checking if this member was added
INFO[0000] [membership] Member was not added
INFO[0000] Removing existing data dir "/var/lib/etcd"
INFO[0000] [membership] Adding member
INFO[0000] [membership] Checking if member was started
INFO[0000] [membership] Member was not started
INFO[0000] [membership] Removing existing data dir "/var/lib/etcd"
INFO[0000] [install] extracting etcd archive /var/cache/etcdadm/etcd/v3.5.13/etcd-v3.5.13-linux-amd64.tar.gz to /tmp/etcd204417867
INFO[0001] [install] verifying etcd 3.5.13 is installed in /opt/bin/
INFO[0010] [health] Checking local etcd endpoint health
INFO[0010] [health] Local etcd endpoint is healthy

清理节点

安装失败，或节点下线时

etcdadm reset

高级用法

从快照还原集群

etcdadm init --snapshot /path/to/etcd.snapshot

使用

$ etcdctl member list --cacert=/etc/etcd/pki/ca.crt --cert=/etc/etcd/pki/server.crt --key=/etc/etcd/pki/server.key
9d4f5cf4152016b2, started, etcd1, https://172.17.0.3:2380, https://172.17.0.3:2379, false
e541d05ce4978fd2, started, etcd2, https://172.17.0.4:2380, https://172.17.0.4:2379, false
ffde685c2fd67496, started, etcd3, https://172.17.0.5:2380, https://172.17.0.5:2379, false

F&Q

etcdadm join 失败

# 新节点日志
INFO[0000] [membership] Checking if this member was added
{"level":"warn","ts":"2024-04-21T11:06:35.761Z","caller":"clientv3/retry_interceptor.go:61","msg":"retrying of unary invoker failed","target":"endpoint://client-40b4d603-dba4-4a01-85a8-a5127ab4bb45/172.17.0.3:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
FATA[0005] [membership] Error listing members: context deadline exceeded

# 被加入的 etcd 日志
$ journalctl -u etcd -f
...
Apr 21 11:19:50 etcd1 etcd[555]: rejected connection from "172.17.0.4:59730" (error "remote error: tls: bad certificate", ServerName "")
...
  • 查看证书信息
openssl x509 .crt ... 
$ openssl x509 -in /etc/etcd/pki/server.crt -noout -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 961923162268170567 (0xd5970050436f947)
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN = etcd
        Validity
            Not Before: Apr 21 10:54:52 2024 GMT
            Not After : Apr 21 10:54:53 2025 GMT
        Subject: CN = etcd1
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (2048 bit)
                Modulus:
                    xxx
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage:
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Authority Key Identifier:
                keyid:3C:53:43:2E:40:CA:BA:67:1F:DC:F0:CF:80:31:C6:36:0B:F5:D7:16

            X509v3 Subject Alternative Name:
                DNS:etcd1, IP Address:172.17.0.3, IP Address:127.0.0.1
    Signature Algorithm: sha256WithRSAEncryption
    xxx

etcdadmin init --server-cert-extra-sans "etcd1,etcd2,etcd3,172.17.0.3,172.17.0.4,172.17.0.5" 指定集群其他节点的信息，之后可以查看到正确的证书信息

openssl x509 -in /etc/etcd/pki/server.crt -noout -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 1973058493428135934 (0x1b61b63efdd9f7fe)
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN = etcd
        Validity
            Not Before: Apr 21 11:15:13 2024 GMT
            Not After : Apr 21 11:15:13 2025 GMT
        ...
        X509v3 extensions:
            ...

            X509v3 Subject Alternative Name:
                DNS:etcd1, DNS:etcd2, DNS:etcd3, DNS:etcd2, IP Address:172.17.0.3, IP Address:172.17.0.4, IP Address:172.17.0.5, IP Address:172.17.0.4, IP Address:127.0.0.1

参考

  1. https://github.com/kubernetes-sigs/etcdadm
最近更新
相关文章
最新评论
Home Archives Categories Tags Statistics
本文总阅读量 次 本站总访问量 次 本站总访客数 人