博信信息网k8s+etcd集群配置(k8s etcd请求超时)
登陆github , https://github.com/kubernetes/release
通过git clone 命令下载到linux， 进入rpm目录，安装好docker执行 docker-build.sh脚本，即可在output目录下编译出rpm包，
1.2 下载镜像由于gci.io 是goolge的， 所以使用kubeadm init时 会出现下载不了镜像，镜像的版本会更具rpm版本不同略有不同。
通过FQ可以把镜像下载到本地， 通过docker push 到私有registry，kubeadm init 修改配置文件使用私有registry，apiserver 会启动不了
查看使用的镜像版本，kubeadm init 会出现/etc/kubernetes/manifests/ 目录，里面的json文件，可以查看image
但是还会有一些镜像无法找到，比如kubedns、pause 等， 可以在源码中找到
https://github.com/kubernetes/kubernetes/blob/master/cmd/kubeadm/app/images/images.go
二.ETCD集群配置ETCD参数说明—data-dir 指定节点的数据存储目录，这些数据包括节点ID，集群ID，集群初始化配置，Snapshot文件，若未指定—wal-dir，还会存储WAL文件；—wal-dir 指定节点的was文件的存储目录，若指定了该参数，wal文件会和其他数据文件分开存储。—name 节点名称—initial-advertise-peer-urls 告知集群其他节点url.— listen-peer-urls 监听URL，用于与其他节点通讯— advertise-client-urls 告知客户端url, 也就是服务的url— initial-cluster-token 集群的ID— initial-cluster 集群中所有节点使用阿里云yum源
192.168.20.206 192.168.20.223 192.168.20.224 三台分别安装 etcd服务器
[root@docker1~]# yum -y install etcd
查看etcd版本
[root@kubernetes ~]# etcdctl -v
etcdctl version: 3.0.15
API version: 2
docker1 /etc/etcd/etcd.conf配置文件
[root@docker1 ~]# grep -v "#" /etc/etcd/etcd.conf
ETCD_NAME=etcd1
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_ADVERTISE_CLIENT_URLS="http://192.168.20.209:2379,http://127.0.0.1:2379"
docker2 /etc/etcd/etcd.conf配置文件
[root@docker2 ~]# grep -v "#" /etc/etcd/etcd.conf
ETCD_NAME=etcd2
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_ADVERTISE_CLIENT_URLS="http://192.168.20.223:2379,http://127.0.0.1:2379"
docker3 /etc/etcd/etcd.conf配置文件
[root@docker3 etcd]# grep -v "#" /etc/etcd/etcd.conf
ETCD_NAME=etcd3
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_ADVERTISE_CLIENT_URLS="http://192.168.20.224:2379,http://127.0.0.1:2379"
修改/usr/lib/systemd/system/etcd.service
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/etcd.conf
User=etcd
# setfailure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
重启3台etcd的服务
[root@docker1 etcd]# systemctl restart etcd.service
[root@docker2 etcd]# systemctl restart etcd.service
[root@docker3 etcd]# systemctl restart etcd.service
etcd集群操作
[root@docker1 ~]# etcdctl member list
b6d60bb2e9e4f37c: name=etcd1 peerURLs=http://192.168.20.209:2380 clientURLs= isLeader=false
c4abafabbe0d2097: name=etcd3 peerURLs=http://192.168.20.224:2380 clientURLs=http://127.0.0.1:2379,http://192.168.20.224:2379 isLeader=false
e8f6dc6436ef6d63: name=etcd2 peerURLs=http://192.168.20.223:2380 clientURLs=http://127.0.0.1:2379,http://192.168.20.223:2379 isLeader=true
cluster
[root@docker1 ~]# etcdctl member remove b6d60bb2e9e4f37c
Removed member b6d60bb2e9e4f37c from cluster
[root@docker1 ~]# etcdctl member list
c4abafabbe0d2097: name=etcd3 peerURLs=http://192.168.20.223:2380 clientURLs=http://127.0.0.1:2379,http://192.168.20.223:2379 isLeader=false
e8f6dc6436ef6d63: name=etcd2 peerURLs=http://192.168.20.224:2380 clientURLs=http://127.0.0.1:2379,http://192.168.20.224:2379 isLeader=true
添加节点：
1. 关闭etcd1 节点的进程,模拟节点挂了
[root@docker1 ~]# systemctl stop etcd.service
2. 192.168.20.209 删除数据
[root@etcd1 ~]# rm /var/lib/etcd/default.etcd/member/* -rf
3.etcd2 添加节点
[root@etcd2 ~]# etcdctl member add etcd1 http://192.168.20.209:2380
Added member named etcd1 with ID d92ef1d9c9d9ebe7 to cluster
ETCD_NAME="etcd1"
ETCD_INITIAL_CLUSTER_STATE="existing"
状态变成 如下：
[root@docker1 ~]# etcdctl member list
c4abafabbe0d2097: name=etcd3 peerURLs=http://192.168.20.224:2380 clientURLs=http://127.0.0.1:2379,http://192.168.20.224:2379 isLeader=false
d92ef1d9c9d9ebe7[unstarted]: peerURLs=http://192.168.20.226:2380
e8f6dc6436ef6d63: name=etcd2 peerURLs=http://192.168.20.225:2380 clientURLs=http://127.0.0.1:2379,http://192.168.20.223:2379 isLeader=true
5. 修改/etc/etcd/etcd.conf ETCD_INITIAL_CLUSTER_STATE 由 new 变成 existing
此时d92ef1d9c9d9ebe7 192.168.20.226 状态为[unstarted]:
6. 重启 etcd1 的服务进程
[root@docker1 ~]# systemctl start etcd.service
7. 查看etcd 服务
[root@docker1 etcd]# etcdctl member list
5190eb3277e38296: name=etcd1 peerURLs=http://192.168.20.226:2380 clientURLs=http://127.0.0.1:2379,http://192.168.20.209:2379 isLeader=false
c4abafabbe0d2097: name=etcd3 peerURLs=http://192.168.20.224:2380 clientURLs=http://127.0.0.1:2379,http://192.168.20.224:2379 isLeader=false
e8f6dc6436ef6d63: name=etcd2 peerURLs=http://192.168.20.225:2380 clientURLs=http://127.0.0.1:2379,http://192.168.20.225:2379 isLeader=true
y ebtables socat四 初始化master1
Ebtables即是以太网桥防火墙,以太网桥工作在数据链路层,Ebtables来过滤数据链路层数据包
启动kubelet客户端
[root@docker1 etcd]# systemctl start kubelet.service
[root@docker1 etcd]# systemctl enable kubelet.service
添加vip
[root@docker1 etcd]# ip addr add 192.168.20.227/24 dev ens32
[root@docker1 etcd]# kubeadm init --api-advertise-addresses=192.168.20.227 --external-etcd-endpoints=http://192.168.20.209:2379,http://192.168.20.223:2379,http://192.168.20.224:2379
Flag --external-etcd-endpoints has been deprecated, this flag will be removed when componentconfig exists
[kubeadm] WARNING: kubeadm is in alpha, please do not use it for production clusters.
[preflight] Running pre-flight checks
[preflight] Starting the kubelet service
[init] Using Kubernetes version: v1.5.3
[tokens] Generated token: "7fd3db.6c8b8f165050c555"
[certificates] Generated Certificate Authority key and certificate.
[certificates] Generated API Server key and certificate
[certificates] Generated Service Account signing keys
[certificates] Created keys and certificates in "/etc/kubernetes/pki"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
[apiclient] Created API client, waiting for the control plane to become ready
[apiclient] All control plane components are healthy after 18.581028 seconds
[apiclient] Waiting for at least one node to register and become ready
[apiclient] First node is ready after 4.009030 seconds
[apiclient] Creating a test deployment
[apiclient] Test deployment succeeded
[token-discovery] Created the kube-discovery deployment, waiting for it to become ready
[token-discovery] kube-discovery is ready after 2.004518 seconds
[addons] Created essential addon: kube-proxy
[addons] Created essential addon: kube-dns
Your Kubernetes master has initialized successfully!
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
http://kubernetes.io/docs/admin/addons/
You can now join any number of machines by running the following on each node:
kubeadm join --token=7fd3db.6c8b8f165050c555 192.168.20.227
--api-advertise-addresses 支持多个ip,但是会导致kubeadm join无法正常加入, 所以对外服务只配置为一个vip
如果使用flannel 网络，使用：
kubeadm init --api-advertise-addresses=192.168.20.227 --external-etcd-endpoints=http://192.168.20.209:2379,http://192.168.20.223:2379,http://192.168.20.224:2379 --pod-network-cidr=10.244.0.0/16
五，部署其他master（docker2 docker3）
安装rpm包（kubeadm, ）六 查看各master 进程、pod情况
[root@docker2 ~]# rpm -ivh kubeadm-1.6.0-0.alpha.0.2074.a092d8e0f95f52.x86_64.rpm kubectl-1.5.1-0.x86_64.rpm kubelet-1.5.1-0.x86_64.rpm kubernetes-cni-0.3.0.1-0.07a8a2.x86_64.rpm
[root@docker2 ~]# systemctl start kubelet.service ; systemctl enable kubelet.service
[root@docker2 ~]# scp -r 192.168.20.209:/etc/kubernetes/* /etc/kubernetes/
kube-controller-manager kube-scheduler 通过 --leader-elect实现了分布式锁. 所以三个master节点可以正常运行
192.168.20.209 master，启动了 192.168.20.227 vip
192.168.20.223 master
192.168.20.224 master
192.168.20.209 查看启动的node节点， 和vip地址
查看master 节点pod 情况， kube-dns pod 没有启动，应为没有配置网络
七 创建POD网络创建好集群后，为了能让容器进行跨主机通讯还要部署 Pod 网络，这里使用官方推荐的 weave 方式，也可以采用 flannel，以下为 weave 示例
只在：docker1上执行。
[root@docker1 ~]#docker pull weaveworks/weave-kube:1.9.2八 采用daemonsets 方式，实现核心组件高可用dns 组件，只在docker 上执行
[root@docker1 ~]#docker tag weaveworks/weave-kube:1.9.2 weaveworks/weave-kube:1.9.2
[root@docker1 ~]#docker rmi weaveworks/weave-kube:1.7.2
[root@docker1 ~]# kubectl create -f weave-kube.yaml
或者 kubectl create -f https://git.io/weave-kube
如果使用 flannel 网络
wget https://github.com/coreos/flannel/blob/master/Documentation/kube-flannel.yml
kubectl create -f kube-flannel.yml
1.方案一，直接replicas
[root@docker1 ~]# kubectl scale deploy/kube-dns --replicas=3 --namespace=kube-system
或者 kubectl scale deployment/kube-dns --replicas=3 --namespace=kube-system
使用最新的 dns
Kubernetes DNS pod拥有3个容器 - kubedns，dnsmasq和一个名为healthz的健康检查。kubedns进程监视Kubernetes主服务和端点的更改，并维护内存中查找结构来服务DNS请求。dnsmasq容器添加DNS缓存以提高性能。healthz容器在执行双重健康检查（对于dnsmasq和kubedns）时提供单个健康检查端点。
1.删除自带dns组件
[root@docker1 ~]# kubectl delete deploy/kube-dns svc/kube-dns -n kube-system
2.下载最新的dns组件
[root@docker1 ~]# cd /root/dns/
for i in Makefile kubedns-controller.yaml.base kubedns-svc.yaml.base transforms2salt.sed transforms2sed.sed; do
wget https://raw.githubusercontent.com/kubernetes/kubernetes/master/cluster/addons/dns/$i
done
[root@docker1 dns]# ls
kubedns-controller.yaml.base kubedns-svc.yaml.base kubernetes-dashboard.yaml Makefile transforms2salt.sed transforms2sed.sed
3. 修改本地环境
[root@docker1 ~]# kubectl get svc |grep kubernetes
kubernetes 10.0.96.1 <none> 443/TCP 6d
[root@docker1 dns]# DNS_SERVER_IP=10.0.96.10
[root@docker1 dns]# DNS_DOMAIN=cluster.local
修改transforms2sed.sed里面2个变量为实际网段及搜索域，修改前如下
[root@k8s-master kubernetes]# cat transforms2sed.sed
s/__PILLAR__DNS__SERVER__/$DNS_SERVER_IP/g
s/__PILLAR__DNS__DOMAIN__/$DNS_DOMAIN/g
/__PILLAR__FEDERATIONS__DOMAIN__MAP__/d
s/__MACHINE_GENERATED_WARNING__/Warning: This is a file generated from the base underscore template file: __SOURCE_FILENAME__/g
修改后如下
[root@k8s-master kubernetes]# cat transforms2sed.sed
s/__PILLAR__DNS__SERVER__/10.254.0.10/g
s/__PILLAR__DNS__DOMAIN__/cluster.local/g
/__PILLAR__FEDERATIONS__DOMAIN__MAP__/d
s/__MACHINE_GENERATED_WARNING__/Warning: This is a file generated from the base underscore template file: __SOURCE_FILENAME__/g
编译
[root@k8s-master kubernetes]# make
sed -f transforms2salt.sed kubedns-controller.yaml.base | sed s/__SOURCE_FILENAME__/kubedns-controller.yaml.base/g > kubedns-controller.yaml.in
sed -f transforms2salt.sed kubedns-svc.yaml.base | sed s/__SOURCE_FILENAME__/kubedns-svc.yaml.base/g > kubedns-svc.yaml.in
sed -f transforms2sed.sed kubedns-controller.yaml.base | sed s/__SOURCE_FILENAME__/kubedns-controller.yaml.base/g > kubedns-controller.yaml.sed
sed -f transforms2sed.sed kubedns-svc.yaml.base | sed s/__SOURCE_FILENAME__/kubedns-svc.yaml.base/g > kubedns-svc.yaml.sed
4，修改RC和SVC文件
#现在编辑文件kubedns-controller.yaml.sed删除（这是用#注释）volume的配置，以避免两个问题：
#1.错误: error validating "kubedns-controller.yaml.sed.bak": error validating data: found invalid field optional for v1.ConfigMapVolumeSource; if you choose to ignore these errors, turn validation off with --validate=false
#2.创建容器后, tail /var/log/messages, 显示错误: configmaps "kube-dns" not found
[root@docker1 ~]# cat kubedns-controller.yaml.sed
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: kube-dns
namespace: kube-system
labels:
k8s-app: kube-dns
kubernetes.io/cluster-service: "true"
spec:
strategy:
rollingUpdate:
maxSurge: 10%
maxUnavailable: 0
selector:
matchLabels:
k8s-app: kube-dns
template:
metadata:
labels:
k8s-app: kube-dns
annotations:
scheduler.alpha.kubernetes.io/critical-pod: ''
scheduler.alpha.kubernetes.io/tolerations: '[{"key":"CriticalAddonsOnly", "operator":"Exists"}]'
spec:
containers:
- name: kubedns
image: gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.12.1
resources:
limits:
memory: 170Mi
requests:
cpu: 100m
memory: 70Mi
livenessProbe:
httpGet:
path: /healthcheck/kubedns
port: 10054
scheme: HTTP
initialDelaySeconds: 60
timeoutSeconds: 5
successThreshold: 1
failureThreshold: 5
readinessProbe:
httpGet:
path: /readiness
port: 8081
scheme: HTTP
initialDelaySeconds: 3
timeoutSeconds: 5
args:
- --domain=cluster.local.
- --dns-port=10053
- --config-dir=/kube-dns-config
- --v=2
env:
- name: PROMETHEUS_PORT
value: "10055"
ports:
- containerPort: 10053
name: dns-local
protocol: UDP
- containerPort: 10053
name: dns-tcp-local
protocol: TCP
- containerPort: 10055
name: metrics
protocol: TCP
- name: dnsmasq
image: gcr.io/google_containers/k8s-dns-dnsmasq-amd64:1.12.1
livenessProbe:
httpGet:
path: /healthcheck/dnsmasq
port: 10054
scheme: HTTP
initialDelaySeconds: 60
timeoutSeconds: 5
successThreshold: 1
failureThreshold: 5
args:
- --cache-size=1000
- --server=/cluster.local/127.0.0.1#10053
- --server=/in-addr.arpa/127.0.0.1#10053
- --server=/ip6.arpa/127.0.0.1#10053
- --log-facility=-
ports:
- containerPort: 53
name: dns
protocol: UDP
- containerPort: 53
name: dns-tcp
protocol: TCP
# see: https://github.com/kubernetes/kubernetes/issues/29055 for details
resources:
requests:
cpu: 150m
memory: 10Mi
- name: sidecar
image: gcr.io/google_containers/k8s-dns-sidecar-amd64:1.12.1
livenessProbe:
httpGet:
path: /metrics
port: 10054
scheme: HTTP
initialDelaySeconds: 60
timeoutSeconds: 5
successThreshold: 1
failureThreshold: 5
args:
- --v=2
- --logtostderr
- --probe=kubedns,127.0.0.1:10053,kubernetes.default.svc.cluster.local,5,A
- --probe=dnsmasq,127.0.0.1:53,kubernetes.default.svc.cluster.local,5,A
ports:
- containerPort: 10054
name: metrics
protocol: TCP
resources:
requests:
memory: 20Mi
cpu: 10m
dnsPolicy: Default # Don't use cluster DNS.
[root@docker1 dns ]# cat kubedns-svc.yaml.sed
apiVersion: v1
kind: Service
metadata:
name: kube-dns
namespace: kube-system
labels:
k8s-app: kube-dns
kubernetes.io/cluster-service: "true"
kubernetes.io/name: "KubeDNS"
spec:
selector:
k8s-app: kube-dns
clusterIP: 10.254.0.10
ports:
- name: dns
port: 53
protocol: UDP
- name: dns-tcp
port: 53
protocol: TCP
[root@docker1 dns]# kubectl create -f kubedns-controller.yaml.sed
deployment "kube-dns" created
[root@docker1 dns]# kubectl create -f kubedns-svc.yaml.sed
service "kube-dns" created
[root@docker1 dns]# kubectl get service --all-namespaces
NAMESPACE NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default kubernetes 10.96.0.1 <none> 443/TCP 59m
kube-system kube-dns 10.96.0.10 <none> 53/UDP,53/TCP 17s
测试创建2个pod
[root@docker1 ~]# cat centos.yaml
apiVersion: v1
kind: Service
metadata:
name: my-centos-svc
labels:
app: centos
spec:
ports:
- port: 80
selector:
app: centos
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: my-centos
spec:
replicas: 1
template:
metadata:
labels:
app: centos
spec:
containers:
- name: centos
image: docker.cinyi.com:443/centos7.3
ports:
- containerPort: 80
[root@docker1 ~]# vim nginx.yaml
apiVersion: v1
kind: Service
metadata:
name: my-nginx-svc
labels:
app: nginx
spec:
ports:
- port: 80
selector:
app: nginx
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: my-nginx
spec:
replicas: 1
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: docker.cinyi.com:443/senyint/nginx
ports:
- containerPort: 80
[root@docker1 ~]# kubectl create -f nginx.yaml -f centos.yaml
方案二、采用采用daemonsets方式,实现核心组件实现高可用，
DaemonSet能够让所有（或者一些特定）的Node节点运行同一个pod。当节点加入到kubernetes集群中，pod会被（DaemonSet）调度到该节点上运行，当节点从kubernetes集群中被移除，被（DaemonSet）调度的pod会被移除
DaemonSet 与 nodeSelector 合用 ，并且在使用的node 打上label，这样就可以指定到不同的node节点上。
[root@docker1 dns]# cat kubends.yaml
apiVersion: v1
kind: Service
metadata:
name: kube-dns
namespace: kube-system
labels:
k8s-app: kube-dns
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcile
kubernetes.io/name: "KubeDNS"
spec:
selector:
k8s-app: kube-dns
clusterIP: 10.96.0.10
ports:
- name: dns
port: 53
protocol: UDP
- name: dns-tcp
port: 53
protocol: TCP
---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
name: kube-dns
namespace: kube-system
labels:
k8s-app: kube-dns
kubernetes.io/cluster-service: "true"
spec:
selector:
matchLabels:
k8s-app: kube-dns
template:
metadata:
labels:
k8s-app: kube-dns
annotations:
scheduler.alpha.kubernetes.io/critical-pod: ''
scheduler.alpha.kubernetes.io/tolerations: '[{"key":"CriticalAddonsOnly", "operator":"Exists"}]'
spec:
nodeSelector:
zone: master
containers:
- name: kubedns
image: gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.12.1
resources:
limits:
memory: 170Mi
requests:
cpu: 100m
memory: 70Mi
livenessProbe:
httpGet:
path: /healthcheck/kubedns
port: 10054
scheme: HTTP
initialDelaySeconds: 60
timeoutSeconds: 5
successThreshold: 1
failureThreshold: 5
readinessProbe:
httpGet:
path: /readiness
port: 8081
scheme: HTTP
initialDelaySeconds: 3
timeoutSeconds: 5
args:
- --domain=cluster.local.
- --dns-port=10053- --v=2
env:
- name: PROMETHEUS_PORT
value: "10055"
ports:
- containerPort: 10053
name: dns-local
protocol: UDP
- containerPort: 10053
name: dns-tcp-local
protocol: TCP
- containerPort: 10055
name: metrics
protocol: TCP
- name: dnsmasq
image: gcr.io/google_containers/k8s-dns-dnsmasq-amd64:1.12.1
livenessProbe:
httpGet:
path: /healthcheck/dnsmasq
port: 10054
scheme: HTTP
initialDelaySeconds: 60
timeoutSeconds: 5
successThreshold: 1
failureThreshold: 5
args:
- --cache-size=1000
- --server=/cluster.local/127.0.0.1#10053
- --server=/in-addr.arpa/127.0.0.1#10053
- --server=/ip6.arpa/127.0.0.1#10053
- --log-facility=-
ports:
- containerPort: 53
name: dns
protocol: UDP
- containerPort: 53
name: dns-tcp
protocol: TCP
resources:
requests:
cpu: 150m
memory: 10Mi
- name: sidecar
image: gcr.io/google_containers/k8s-dns-sidecar-amd64:1.12.1
livenessProbe:
httpGet:
path: /metrics
port: 10054
scheme: HTTP
initialDelaySeconds: 60
timeoutSeconds: 5
successThreshold: 1
failureThreshold: 5
args:
- --v=2
- --logtostderr
- --probe=kubedns,127.0.0.1:10053,kubernetes.default.svc.cluster.local,5,A
- --probe=dnsmasq,127.0.0.1:53,kubernetes.default.svc.cluster.local,5,A
ports:
- containerPort: 10054
name: metrics
protocol: TCP
resources:
requests:
memory: 20Mi
cpu: 10m
dnsPolicy: Default # Don't use cluster DNS.
创建 dns service 和 daemonset ，创建完成 没有dns pod的信息，需要告诉在那个节点启动，使用 kubecet label 命令
[root@docker1 dns]# kubectl create -f /root/dns/kubends.yaml
#zone=master，在dns.yaml文件的 nodeSelector 定义
[root@docker1 dns]# kubectl label node docker1 zone=master
执行完成后，只在docker1 上启动dns pod，为了冗余，需要在 docker2 docker3 也启动
[root@docker2 ~]# kubectl label node docker1 zone=master
[root@docker3 ~]# kubectl label node docker1 zone=master
kube-discovery，kube-discovery 主要负责集群密钥的分发,如果这个组件不正常, 将无法正常新增节点kubeadm joindocker1上直接执行
方案一,直接添加副本数，出现错误，只能存在唯一的 一个node上
kubectl scale deploy/kube-discovery --replicas=3 -n kube-system
报错信息为：
pod (kube-discovery-1769846148-55sjr) failed to fit in any node fit failure summary on nodes : MatchNodeSelector (2), PodFitsHostPorts (1)
方案二
#1. 导出kube-discovery配置
kubectl get deploy/kube-discovery -n kube-system -o yaml > /data/kube-discovery.yaml
#2. 把Deployment类型改为DaemonSet,并加上master nodeSelector
#3. 删掉自带kube-discovery
kubectl delete deploy/kube-discovery -n kube-system
#4. 部署kube-discovery
kubectl apply -f kube-discovery.yaml
#5. 打docker1 docker2 docker3 标记
[root@docker1 ~]# kubectl label node docker1 role=discovery
[root@docker2 ~]# kubectl label node docker2 role=discovery
[root@docker3 ~]# kubectl label node docker3 role=discovery
#6.配置中删除了 annotaions 选项，不然只能启动一个descovery pod
注意： nodeSelector 标记变成了 zone=descovery
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
annotations:
deployment.kubernetes.io/revision: "1"
creationTimestamp: 2017-03-09T10:07:44Z
generation: 1
labels:
component: kube-discovery
k8s-app: kube-discovery
kubernetes.io/cluster-service: "true"
name: kube-discovery
tier: node
name: kube-discovery
namespace: kube-system
resourceVersion: "85792"
selfLink: /apis/extensions/v1beta1/namespaces/kube-system/deployments/kube-discovery
uid: 3d1333e4-04b0-11e7-ae57-0050568450f4
spec:
selector:
matchLabels:
component: kube-discovery
k8s-app: kube-discovery
kubernetes.io/cluster-service: "true"
name: kube-discovery
tier: node
template:
metadata:
creationTimestamp: null
labels:
component: kube-discovery
k8s-app: kube-discovery
kubernetes.io/cluster-service: "true"
name: kube-discovery
tier: node
spec:
nodeSelector:
role: discovery
containers:
- command:
- /usr/local/bin/kube-discovery
image: gcr.io/google_containers/kube-discovery-amd64:1.0
imagePullPolicy: IfNotPresent
name: kube-discovery
ports:
- containerPort: 9898
hostPort: 9898
name: http
protocol: TCP
resources: {}
securityContext:
seLinuxOptions:
type: spc_t
terminationMessagePath: /dev/termination-log
volumeMounts:
- mountPath: /tmp/secret
name: clusterinfo
readOnly: true
dnsPolicy: ClusterFirst
hostNetwork: true
restartPolicy: Always
securityContext: {}
terminationGracePeriodSeconds: 30
volumes:
- name: clusterinfo
secret:
defaultMode: 420
secretName: clusterinfo
到目前为止,三个master节点 相互独立运行,互补干扰.kube-apiserver作为核心入口, 可以使用keepalived 实现高可用,kubeadm join暂时不支持负载均衡的方式
3台mater安装keepalived
yum -y install keepalived
以下是三台服务器的keepalived 配置文件
docker1 keepalived 配置文件
global_defs {
router_id LVS_k8s
}
vrrp_script CheckK8sMaster {
script "curl -k https://192.168.20.227:6443"
interval 3
timeout 9
fall 2
rise 2
}
vrrp_instance VI_1 {
state MASTER
interface ens32
virtual_router_id 61
priority 115
nopreempt
authentication {
auth_type PASS
auth_pass 123456
}
unicast_peer {
192.168.20.223
192.168.20.224
}
virtual_ipaddress {
192.168.20.227/24
}
track_script {
CheckK8sMaster
}
}
docker2 keepalived 配置文件
global_defs {
router_id LVS_k8s
}
vrrp_script CheckK8sMaster {
script "curl -k https://192.168.20.227:6443"
interval 3
timeout 9
fall 2
rise 2
}
vrrp_instance VI_1 {
state BACKUP
interface ens32
virtual_router_id 61
priority 110
nopreempt
authentication {
auth_type PASS
auth_pass 123456
}
unicast_peer {
192.168.20.224
192.168.20.209
}
virtual_ipaddress {
192.168.20.227/24
}
track_script {
CheckK8sMaster
}
}
docker3 keeplived 配置文件十 验证
global_defs {
router_id LVS_k8s
}
vrrp_script CheckK8sMaster {
script "curl -k https://192.168.20.227:6443"
interval 3
timeout 9
fall 2
rise 2
}
vrrp_instance VI_1 {
state MASTER
interface ens32
virtual_router_id 61
priority 100
nopreempt
authentication {
auth_type PASS
auth_pass 123456
}
unicast_peer {
192.168.20.223
192.168.20.209
}
virtual_ipaddress {
192.168.20.227/24
}
track_script {
CheckK8sMaster
}
}
加入节点
[root@docker4 ~]# yum install -y ebtables socat docker
[root@docker4 ~]# scp 192.168.20.209:/root.*.rpm /root/
[root@docker4 ~]# ls
kubectl-1.5.1-0.x86_64.rpm kubeadm-1.6.0-0.alpha.0.2074.a092d8e0f95f52.x86_64.rpm kubelet-1.5.1-0.x86_64.rpm kubernetes-cni-0.3.0.1-0.07a8a2.x86_64.rpm
[root@docker4 ~]# rpm -ivh *.rpm
[root@docker4 ~]# systemctl start kubelet.service
[root@docker4 ~]# systemctl enable kubelet.service
[root@docker4 ~]# kubeadm join --token=7fd3db.6c8b8f165050c555 192.168.20.227
登陆到 docker1 上查看 docker4 是否加入到集群
192.168.20.209 master docker1 关机
192.168.20.223 /var/log/messages 日志，192.168.20.227 自动添加
master1 docker 服务器关机， 所有docker节点状态都为丢失，使用docker2，登陆到pod中， 解析可以访问。
kubernetes 安装dashboard[root@docker2 ~]# wget https://rawgit.com/kubernetes/dashboard/master/src/deploy/kubernetes-dashboard.yaml
[root@docker2 ~]# kubectl create -f kubernetes-dashboard.yaml
[root@docker2 ~]#kubectl describe service kubernetes-dashboard --namespace=kube-system
IE访问：
http://192.168.20.227:31766
安装headper插件方便监控流量图Heapster:对集群中各node上cAdvisor的数据采集汇聚的系统，通过访问node上kubelet的API，在通过kubelet调用cAdvisor的API来采集该节点上所有容器的性能数据，
heapster对性能数据进行聚合，并将结果保存到后端存储系统中，Headper支持多种后端存储系统，包括memory ，InfluxDB, BigQuery, 目前Heapster项目主页为
https://github.com/kubernetes/heapster
InfluxDB: 是分布式时序数据库，每条记录都带有时间戳属性，主要用于实时数据采集、事件跟踪记录、存储时间图表，原始数据等。 InfluxDB 提供了 RESt API 用于数据
的存储和查询， 主页是 http://influxdb.com
Grafana: 通过dashboard 将influxDB中的实时数据展现成图表或者曲线等形式， 主页是 http://grafana.org
[root@docker2 ~]# git clone https://github.com/kubernetes/heapster
[root@docker2 ~]# cd /root/heapster/deploy/kube-config/influxdb/
[root@docker2 ~]# ls
grafana-service.yaml heapster-controller.yaml heapster-service.yaml influxdb-grafana-controller.yaml influxdb-service.yaml
由于apiserver 使用的vip，并且是容器化后，默认是127.0.0.1：8080 端口， heapster-controller.yaml文件需要连接apiserver：8080端口， telnet 192.168.20.227 超时,所以使用
loadbalancer IP地址，前提是dns能够正常解析。其实 10.96.0.1 就是apiserver的服务端口，端口为443
登录到nginx pod中，测试dns解析结果
修改 heapster-controller.yaml 文件，注意 dns域名要写完整。kubernetes.default.svc.cluster.local
spec:
containers:
- name: heapster
image: kubernetes/heapster:canary
imagePullPolicy: IfNotPresent
command:
- /heapster
- --source=kubernetes:https://kubernetes.default.svc.cluster.local
- --sink=influxdb:http://monitoring-influxdb.kube-system.svc.cluster.local:8086
[root@docker2 ~]# kubectl create -f /root/heapster/deploy/kube-config/influxdb/Kubernetes Nginx Ingress
登录dashboard 映射的nodeport端口，查看监控是否正常，端口为30263.
Kubernetes 暴露服务的方式目前只有三种：LoadBlancer Service、NodePort Service、Ingress
由于pod 和 service 是 kubernetes 集群范围内的虚拟的概念，所以集群外的客户端无法通过pod ip地址或者service的虚拟IP地址和虚拟端口号访问它们，可以通过将pod 或者service
的端口号映射到宿主机。
1. 将容器应用的端口号映射到物理机
通过设置pod级别的hostNetwork=true,将pod端口映射到物理机上,在容器的ports定义部分如果不指定hostPort，默认hostPort 等于containerPort，如果指定了hostPort，则hostPort必须等于containerPort的值
apiVersion: v1
kind: Pod
metadata:
name: webapp
labels:
app: webapp
spec:
hostNetwork: true
containers:
- name: webapp
image: docker.cinyi.com:443/senyint/nginx
ports:
- containerPort: 80
hostPort: 80
在docker4 上查看端口
2.将service端口映射到物理机
通过设置nodePort映射到物理机上，如果不指定nodeport 将随机分配 30001---30761之间的端口。
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: tomcat-deployment
spec:
replicas: 3
template:
metadata:
labels:
app: tomcat
tier: frontend
spec:
containers:
- name: tomcat
image: docker.cinyi.com:443/tomcat
ports:
- containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: tomcat-server
spec:
type: NodePort
ports:
- port: 11111
targetPort: 8080
nodePort: 30001
selector:
tier: frontend
Ingress 就是能利用 Nginx 方向代理 service 名字和端口，Ingress将不同的URl访问请求转发到后端不同的service，实现http层的业务路由机制，
Ingress的实现需要通过Ingress的定义与Ingress Controller的定义结合起来，才能形成完整的http负载转发功能。
使用Ingress 解决2个问题：1. pod 迁移问题，Pod挂掉时自动从其他机器启动一个新的，所以pod的ip地址发生变化，
2.采用nodeport 暴露server的 nodeport端口，会出现物理机的 端口原来越多，不方便管理。
安装 Ingress Nginx1.部署默认后端，如果访问的域名不存在，全部转到默认后端
地址：https://github.com/kubernetes/ingress/tree/master/examples/deployment/nginx
[root@docker1]# cat /root/default-backend.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: default-http-backend
labels:
k8s-app: default-http-backend
namespace: kube-system
spec:
replicas: 1
template:
metadata:
labels:
k8s-app: default-http-backend
spec:
terminationGracePeriodSeconds: 60
containers:
- name: default-http-backend
image: gcr.io/google_containers/defaultbackend:1.0
livenessProbe:
httpGet:
path: /healthz
port: 8080
scheme: HTTP
initialDelaySeconds: 30
timeoutSeconds: 5
ports:
- containerPort: 8080
resources:
limits:
cpu: 10m
memory: 20Mi
requests:
cpu: 10m
memory: 20Mi
---
apiVersion: v1
kind: Service
metadata:
name: default-http-backend
namespace: kube-system
labels:
k8s-app: default-http-backend
spec:
ports:
- port: 80
targetPort: 8080
selector:
k8s-app: default-http-backend
[root@docker1~ ]# kubcectl create -f default-backend.yaml
2. 部署 Ingress Controller
我在 nginx-ingress-controller.yaml 文件的 kind 使用了DaemonSet，通过nodeselect zone属性，让它在3台master （apiserver）运行，
apiVersion: extensions/v1beta1
kind: DaemonSet
#kind: Deployment
metadata:
name: nginx-ingress-controller
labels:
k8s-app: nginx-ingress-controller
namespace: kube-system
spec:
# replicas: 1
template:
metadata:
labels:
k8s-app: nginx-ingress-controller
spec:
nodeSelector:
zone: nginx-ingress
hostNetwork: true
terminationGracePeriodSeconds: 60
containers:
- image: gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.2
name: nginx-ingress-controller
readinessProbe:
httpGet:
path: /healthz
port: 10254
scheme: HTTP
livenessProbe:
httpGet:
path: /healthz
port: 10254
scheme: HTTP
initialDelaySeconds: 10
timeoutSeconds: 1
ports:
- containerPort: 80
hostPort: 80
- containerPort: 443
hostPort: 443
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
args:
- /nginx-ingress-controller
- --default-backend-service=$(POD_NAMESPACE)/default-http-backend
[root@docker1 ~]# kubectl create -f nginx-ingress-controller.yaml
指定运行的机器,如果已经运行到了其他node节点，使用 kubectl delete pods pod名称 删除。
[root@docker1 ~]# kubectl lable node docker1 zone=nginx-ingress
[root@docker2 ~]# kubectl lable node docker2 zone=nginx-ingress
[root@docker3 ~]# kubectl lable node docker3 zone=nginx-ingress
部署Ingress， 查看所有的service
编写yaml文件，文件内容是域名 - 服务 - 端口
[root@docker1 ~]# vim show-ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: dashboard-nginx-ingress
namespace: kube-system
spec:
rules:
- host: dashboard.feng.com
http:
paths:
- backend:
serviceName: kubernetes-dashboard
servicePort: 80
- host: nginx.feng.com
http:
paths:
- backend:
serviceName: my-nginx-svc
servicePort: 80
[root@docker1~]# kubectl create -f show-ingress.yaml
[root@docker1 ~]# kubectl get ingress --namespace=kube-system -o wide
NAME HOSTS ADDRESS PORTS AGE
dashboard-nginx-ingress dashboard.feng.com,nginx.feng.com 192.168.20.209,192.168.20.223,192.168.20.224 80 19h
2个域名，指定到3个IP上，映射到宿主机的80端口上。
如果是不同namespaces 命名空间下的服务， 可以写2个 ingress yaml文件
[root@docker1 ~]# cat dashboard-ingressA.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: dashboard-ingressa
namespace: kube-system
spec:
rules:
- host: dashboard.feng.com
http:
paths:
- backend:
serviceName: kubernetes-dashboard
servicePort: 80
- host: nginx.feng.com
http:
paths:
- backend:
serviceName: monitoring-grafana
servicePort: 80
[root@docker1 ~]# cat dashboard-ingressB.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: dashboard-ingressb
namespace: test
spec:
rules:
- host: imweb.feng.com
http:
paths:
- backend:
serviceName: im-web
servicePort: 80
[root@docker1 ~]# kubectl create -f default-backend.yaml -f nginx-ingress-controller.yaml -f dashboard-ingressA.yaml -f dashboard-ingressB.yaml
deployment "default-http-backend" created
service "default-http-backend" created
daemonset "nginx-ingress-controller" created
ingress "dashboard-ingressa" created
ingress "dashboard-ingressb"
[root@docker1 ~]# kubectl get ingress --all-namespaces
NAMESPACE NAME HOSTS ADDRESS PORTS AGE
kube-system dashboard-ingressa dashboard.feng.com,nginx.feng.com 80 27s
test dashboard-ingressb imweb.feng.com 80
修改client host后，访问域名
想不使用 Ingress， 直接使用nginx， 设置hostNetwork=true，创建pod和service后，进入到pod中，nslookup 不能解析其他service 的域名。
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: nginx
spec:
replicas: 3
template:
metadata:
labels:
app: nginx
spec:
hostNetwork: true
containers:
- name: nginx
image: docker.cinyi.com:443/senyint/nginx
ports:
- containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: nginx
spec:
ports:
- port: 80
selector:
app: nginx
版权声明
本文仅代表作者观点，不代表博信信息网立场。
作者文章
- 拨号VPS需要手动换IP吗？ 11个月前 (06-25)
- 如何搭建VPS远程拨号服务器？ 11个月前 (06-25)
- 如何为拨号VPS搭建最佳代理？ 11个月前 (06-25)
- 如何利用VPS进行拨号连接？ 11个月前 (06-25)
- 日本VPS拨号动态IP：解锁网络新境界 11个月前 (06-25)