centos7.3.1611部署k8s1.5.2集群-凯发app官方网站

下列文档简单的系统的测试了k8s 1.5.x系列：包括部署集群、创建pod、域名解析、仪表盘、监控、反向代理、存储、日志，另外双向认证自己建证书不太实用就没有列出。本系列文档环境部署使用二进制程序绿色安装，适用于1.5.2、1.5.3、1.5.4及后续版本，只是记得随时更新github上样例url即可。

k8s集群安装部署

http://jerrymin.blog.51cto.com/3002256/1898243

k8s集群rc、svc、pod部署

http://jerrymin.blog.51cto.com/3002256/1900260

k8s集群组件kubernetes-dashboard和kube-dns部署

http://jerrymin.blog.51cto.com/3002256/1900508

k8s集群监控组件heapster部署

http://jerrymin.blog.51cto.com/3002256/1904460

k8s集群反向代理负载均衡组件部署

http://jerrymin.blog.51cto.com/3002256/1904463

k8s集群挂载volume之nfs

http://jerrymin.blog.51cto.com/3002256/1906778

k8s集群挂载volume之glusterfs

http://jerrymin.blog.51cto.com/3002256/1907274

k8s集群日志收集elk架构

http://jerrymin.blog.51cto.com/3002256/1907282

架构

k8s-master 安装etcd，kubernetes-server/client

k8s-node1 安装docker,kubernetes-node/client，flannel

k8s-node2 安装docker,kubernetes-node/client，flannel

一，yum安装的版本如下

centos7.3.1611 yum安装

kubernetes-1.4.0-0.1.git87d9d8d.el7

会安装kubernets-master,node,client及其相关依赖项

kubernetes-master-1.4.0-0.1.git87d9d8d.el7

会产生三个二进制程序kube-apiserver kube-controller-manager kube-scheduler

kubernetes-node-1.4.0-0.1.git87d9d8d.el7

会安装很多依赖包包括docker-1.12.5-14.el7.centos，会安装kubelet kube-proxy

kubernetes-client-1.4.0-0.1.git87d9d8d.el7

会产生一个二进制程序kubectl

kubernetes-unit-test-1.4.0-0.1.git87d9d8d.el7

会安装很多依赖包包括etcd-3.0.15-1.el7，golang，gcc，glibc，rsync等

flannel-0.5.5-2.el7

会产生一个二进制程序flannel

二，本文选择二进制包版本安装最新版测试

github地址：

etct:

flannel:

kubernetes:

docker:

k8s 1.5.2

会产生11个二进制程序hyperkube kubectl kubelet kube-scheduler kubeadm kube-controller-manager kube-discovery kube-proxy kube-apiserver kube-dns kubefed

会产生两个二进制程序kube-proxy kubefed

etcd 3.1.10

/download/v3.1.0/etcd-v3.1.0-linux-amd64.tar.gz

docker 1.13.1

flannel

/download/v0.7.0/flannel-v0.7.0-linux-amd64.tar.gz

三，部署环境

1，准备工作

1），系统最小化安装,然后yum update,升级到最新版本centos7.3.1611

2），设置hostname及hosts

[root@k8s-master ~]# cat /etc/hosts

127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4

::1 localhost localhost.localdomain localhost6 localhost6.localdomain6

172.17.3.20 k8s-master

172.17.3.7 k8s-node1

172.17.3.8 k8s-node2

3），校对时间

[root@k8s-master ~]# ntpdate ntp1.aliyun.com &&hwclock -w

4），关闭selinux及防火墙

[root@k8s-master ~]# sed -i s'/selinux=enforcing/selinux=disabled/g' /etc/sysconfig/selinux

[root@k8s-master ~]# systemctl disable firewalld; systemctl stop firewalld

removed symlink /etc/systemd/system/dbus-org.fedoraproject.firewalld1.service.

removed symlink /etc/systemd/system/basic.target.wants/firewalld.service.

5），重启服务器

2，master节点部署

1），部署etcd服务（目前单点）

[root@k8s-master ~]# tar zxvf etcd-v3.1.0-linux-amd64.tar.gz -c /usr/local/

[root@k8s-master ~]# mv /usr/local/etcd-v3.1.0-linux-amd64/ /usr/local/etcd

[root@k8s-master ~]# ln -s /usr/local/etcd/etcd /usr/local/bin/etcd

[root@k8s-master ~]# ln -s /usr/local/etcd/etcdctl /usr/local/bin/etcdctl

设置systemd服务文件/usr/lib/systemd/system/etcd.service

[unit]

description=eted server

after=network.target

[service]

workingdirectory=/data/etcd/

environmentfile=-/etc/etcd/etcd.conf

execstart=/usr/local/bin/etcd

type=notify

restart=on-failure

limitnofile=65536

[install]

wantedby=multi-user.target

其中workingdirector表示etcd数据保存的目录，需要在启动etcd服务之前进行创建

etcd单点默认配置

[root@k8s-master ~]# cat /etc/etcd/etcd.conf

etcd_name=k8s1

etcd_data_dir="/data/etcd"

etcd_listen_client_urls=""

etcd_advertise_client_urls=""

etcd服务启动

[root@k8s-master ~]# systemctl daemon-reload

[root@k8s-master ~]# systemctl enable etcd.service

[root@k8s-master ~]# systemctl start etcd.service

etcd服务检查

[root@k8s-master ~]# etcdctl cluster-health

member 869f0c691c5458a3 is healthy: got healthy result from

cluster is healthy

[root@k8s-master ~]# etcdctl member list

869f0c691c5458a3: name=k8s1 peerurls= clienturls= isleader=true

2）部署kube-apiserver服务

安装kube-apiserver

[root@k8s-master ~]# tar zxvf kubernetes-server-linux-amd64.tar.gz -c /usr/local/

[root@k8s-master ~]# ln -s /usr/local/kubernetes/server/bin/kube-apiserver /usr/local/bin/kube-apiserver

其他服务顺便做下软链接

[root@k8s-master ~]# ln -s /usr/local/kubernetes/server/bin/hyperkube /usr/local/bin/hyperkube

[root@k8s-master ~]# ln -s /usr/local/kubernetes/server/bin/kubeadm /usr/local/bin/kubeadm

[root@k8s-master ~]# ln -s /usr/local/kubernetes/server/bin/kube-controller-manager /usr/local/bin/kube-controller-manager

[root@k8s-master ~]# ln -s /usr/local/kubernetes/server/bin/kubectl /usr/local/bin/kubectl

[root@k8s-master ~]# ln -s /usr/local/kubernetes/server/bin/kube-discovery /usr/local/bin/kube-discovery

[root@k8s-master ~]# ln -s /usr/local/kubernetes/server/bin/kube-dns /usr/local/bin/kube-dns

[root@k8s-master ~]# ln -s /usr/local/kubernetes/server/bin/kubefed /usr/local/bin/kubefed

[root@k8s-master ~]# ln -s /usr/local/kubernetes/server/bin/kubelet /usr/local/bin/kubelet

[root@k8s-master ~]# ln -s /usr/local/kubernetes/server/bin/kube-proxy /usr/local/bin/kube-proxy

[root@k8s-master ~]# ln -s /usr/local/kubernetes/server/bin/kube-scheduler /usr/local/bin/kube-scheduler

配置kubernetes system config

[root@k8s-master ~]# cat /etc/kubernetes/config

kube_logtostderr="--logtostderr=false"

kube_log_dir="--log-dir=/data/logs/kubernetes"

kube_log_level="--v=2"

kube_allow_priv="--allow-privileged=false"

kube_master="--master="

设置systemd服务文件/usr/lib/systemd/system/kube-apiserver.service

[unit]

description=kubernetes api server

documentation=

after=network.target

after=etcd.service

[service]

environmentfile=-/etc/kubernetes/config

environmentfile=-/etc/kubernetes/apiserver

execstart=/usr/local/bin/kube-apiserver \

$kube_logtostderr \

$kube_log_level \

$kube_etcd_servers \

$kube_api_address \

$kube_api_port \

$kubelet_port \

$kube_allow_priv \

$kube_service_addresses \

$kube_admission_control \

$kube_api_args

restart=on-failure

type=notify

limitnofile=65536

[install]

wantedby=multi-user.target

配置kuber-apiserver启动参数

[root@k8s-master ~]# cat /etc/kubernetes/apiserver

kube_api_address="--insecure-bind-address=0.0.0.0"

kube_etcd_servers="--etcd-servers="

kube_service_addresses="--service-cluster-ip-range=10.254.0.0/16"

kube_admission_control="--admission-control=namespacelifecycle,namespaceexists,limitranger,securitycontextdeny,serviceaccount,resourcequota"

kube_api_args=" "

启动kube-api-servers服务

[root@k8s-master ~]# systemctl daemon-reload

[root@k8s-master ~]# systemctl enable kube-apiserver.service

created symlink from /etc/systemd/system/multi-user.target.wants/kube-apiserver.service to /usr/lib/systemd/system/kube-apiserver.service.

[root@k8s-master ~]# systemctl start kube-apiserver.service

验证服务

3）部署kube-controller-manager服务

设置systemd服务文件/usr/lib/systemd/system/kube-controller-manager.service

[unit]

description=kubernetes controller manager

documentation=

after=kube-apiserver.service

requires=kube-apiserver.service

[service]

environmentfile=-/etc/kubernetes/config

environmentfile=-/etc/kubernetes/controller-manager

execstart=/usr/local/bin/kube-controller-manager \

$kube_logtostderr \

$kube_log_level \

$kube_log_dir \

$kube_master \

$kube_controller_manager_args

restart=on-failure

limitnofile=65536

[install]

wantedby=multi-user.target

配置kube-controller-manager启动参数

[root@k8s-master ~]# cat /etc/kubernetes/controller-manager

kube_controller_manager_args=""

启动kube-controller-manager服务

[root@k8s-master ~]# systemctl daemon-reload

[root@k8s-master ~]# systemctl enable kube-controller-manager

created symlink from /etc/systemd/system/multi-user.target.wants/kube-controller-manager.service to /usr/lib/systemd/system/kube-controller-manager.service.

[root@k8s-master ~]# systemctl start kube-controller-manager

4）部署kube-scheduler服务

设置systemd服务文件/usr/lib/systemd/system/kube-scheduler.service

[unit]

description=kubernetes scheduler plugin

documentation=

after=kube-apiserver.service

requires=kube-apiserver.service

[service]

environmentfile=-/etc/kubernetes/config

environmentfile=-/etc/kubernetes/scheduler

execstart=/usr/local/bin/kube-scheduler \

$kube_logtostderr \

$kube_log_level \

$kube_log_dir \

$kube_master \

$kube_scheduler_args

restart=on-failure

limitnofile=65536

[install]

wantedby=multi-user.target

配置kube-schedulerr启动参数

[root@k8s-master ~]# cat /etc/kubernetes/schedulerr

kube_scheduler_args=""

启动kube-scheduler服务

[root@k8s-master ~]# systemctl daemon-reload

[root@k8s-master ~]# systemctl enable kube-scheduler

created symlink from /etc/systemd/system/multi-user.target.wants/kube-scheduler.service to /usr/lib/systemd/system/kube-scheduler.service.

[root@k8s-master ~]# systemctl start kube-scheduler

2，node节点部署

1）安装docker(或者yum instll docker)

[root@k8s-node1 ~]# tar zxvf docker-1.13.1.tgz -c /usr/local

这里默认docker安装并启动，方便后面测试

[root@k8s-node1 ~]# systemctl start docker.service

2）安装kubernetes客户端

安装kubelet，kube-proxy

[root@k8s-master ~]# tar zxvf kubernetes-client-linux-amd64.tar.gz -c /usr/local/

[root@k8s-node1 ~]# ln -s /usr/local/kubernetes/client/bin/kubectl /usr/local/bin/kubectl

[root@k8s-node1 ~]# ln -s /usr/local/kubernetes/client/bin/kubefed /usr/local/bin/kubefed

kube-proxy包默认client没有可以从server拷贝过来

[root@k8s-node1 ~]# ln -s /usr/local/kubernetes/client/bin/kube-proxy /usr/local/bin/kube-proxy

[root@k8s-node1 ~]# ln -s /usr/local/kubernetes/client/bin/kubelet /usr/local/bin/kubelet

3）部署kubelet服务

配置kubernetes system config

[root@k8s-node1 ~]# cat /etc/kubernetes/config

kube_logtostderr="--logtostderr=false"

kube_log_dir="--log-dir=/data/logs/kubernetes"

kube_log_level="--v=2"

kube_allow_priv="--allow-privileged=false"

kube_master="--master="

设置systemd服务文件/usr/lib/systemd/system/kubelet.service

[unit]

description=kubernetes kubelet server

documentation=

after=docker.service

requires=docker.service

[service]

workingdirectory=/data/kubelet

environmentfile=-/etc/kubernetes/config

environmentfile=-/etc/kubernetes/kubelet

execstart=/usr/local/bin/kubelet \

$kube_logtostderr \

$kube_log_level \

$kube_log_dir \

$kubelet_api_server \

$kubelet_address \

$kubelet_port \

$kubelet_hostname \

$kube_allow_priv \

$kubelet_pod_infra_container \

$kubelet_args

restart=on-failure

[install]

wantedby=multi-user.target

配置kubelet启动参数

[root@k8s-node1 ~]# cat /etc/kubernetes/kubelet

kubelet_address="--address=0.0.0.0"

kubelet_port="--port=10250"

kubelet_hostname="--hostname-override=k8s-node1"

kubelet_api_server="--api-servers="

kubelet_pod_infra_container="--pod-infra-container-image=registry.access.redhat.com/rhel7/pod-infrastructure:latest"

kubelet_args=""

启动kubelet服务

[root@k8s-node1 ~]# systemctl daemon-reload

[root@k8s-node1 ~]# systemctl enable kubelet.service

created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /usr/lib/systemd/system/kubelet.service.

[root@k8s-node1 ~]# systemctl start kubelet.service

4），部署kube-proxy服务

设置systemd服务文件/usr/lib/systemd/system/kube-proxy.service

[unit]

description=kubernetes kube-proxy server

documentation=

after=network.target

[service]

environmentfile=-/etc/kubernetes/config

environmentfile=-/etc/kubernetes/proxy

execstart=/usr/local/bin/kube-proxy \

$kube_logtostderr \

$kube_log_level \

$kube_log_dir \

$kube_master \

$kube_proxy_args

restart=on-failure

limitnofile=65536

[install]

wantedby=multi-user.target

配置kubelet启动参数

[root@k8s-node1 ~]# cat /etc/kubernetes/proxy

kube_proxy_args=""

启动kubelet服务

[root@k8s-node1 ~]# systemctl daemon-reload

[root@k8s-node1 ~]# systemctl enable kube-proxy.service

created symlink from /etc/systemd/system/multi-user.target.wants/kube-proxy.service to /usr/lib/systemd/system/kube-proxy.service.

[root@k8s-node1 ~]# systemctl start kube-proxy.service

验证节点是否启动

[root@k8s-node1 ~]# kubectl get nodes

name status age

k8s-node1 ready 9m

3，配置网络

1），配置etcd

[root@k8s-master ~]# etcdctl set /k8s/network/config '{ "network": "10.1.0.0/16" }'

{ "network": "10.1.0.0/16" }

[root@k8s-master ~]# etcdctl get /k8s/network/config

{ "network": "10.1.0.0/16" }

2)，安装flannel

[root@k8s-node1 ~]# tar zxvf flannel-v0.7.0-linux-amd64.tar.gz -c /usr/local/flannel

[root@k8s-node1 ~]# ln -s /usr/local/flannel/flannel /usr/local/bin/flanneld

[root@k8s-node1 ~]# ln -s /usr/local/flannel/mk-docker-opts.sh /usr/local/bin/mk-docker-opts.sh

3），配置flannel（配置比较麻烦，start脚本和启动脚本参考yum安装时生成的配置）

设置systemd服务文件/usr/lib/systemd/system/flanneld.service

[unit]

description=flanneld overlay address etcd agent

after=network.target

after=network-online.target

wants=network-online.target

after=etcd.service

before=docker.service

[service]

type=notify

environmentfile=/etc/sysconfig/flanneld

environmentfile=-/etc/sysconfig/docker-network

execstart=/usr/local/bin/flanneld-start $flannel_options

execstartpost=/usr/local/bin/mk-docker-opts.sh -k docker_network_options -d /run/flannel/docker

restart=on-failure

[install]

wantedby=multi-user.target

requiredby=docker.service

其中flanneld-start为

[root@k8s-node1 ~]# cat /usr/local/bin/flanneld-start

#!/bin/sh

exec /usr/local/bin/flanneld \

-etcd-endpoints=${flannel_etcd_endpoints:-${flannel_etcd}} \

-etcd-prefix=${flannel_etcd_prefix:-${flannel_etcd_key}} \

"$@"

编辑flannel，设置etcd相关信息

[root@k8s-node1 ~]# cat /etc/sysconfig/flanneld

flannel_etcd_endpoints=""

flannel_etcd_prefix="/k8s/network"

4），启动flannel

注意启动flannel前要关闭docker这样flannel才会覆盖docker0网桥

[root@k8s-node1 ~]# systemctl daemon-reload

[root@k8s-node1 ~]# systemctl enable flanneld.service

[root@k8s-node1 ~]# systemctl stop docker.service

[root@k8s-node1 ~]# systemctl start flanneld.service

flanneld服务启动后就会根据etcd里面配置划分子网了，划分子网是给docker使用的，docker想使用还得折腾一翻，其实就是想办法把几个重要变量传过去，使docker启动时能够使用

注意启动docker前要使某些变量生效，需要source /run/flannel/docker source /run/flannel/subnet.env

[root@k8s-node1 ~]# cat /run/flannel/docker

docker_opt_bip="--bip=10.1.89.1/24"

docker_opt_ipmasq="--ip-masq=true"

docker_opt_mtu="--mtu=1472"

docker_network_options=" --bip=10.1.89.1/24 --ip-masq=true --mtu=1472"

[root@k8s-node1 bin]# cat /run/flannel/docker

docker_opt_bip="--bip=10.1.89.1/24"

docker_opt_ipmasq="--ip-masq=true"

docker_opt_mtu="--mtu=1472"

docker_network_options=" --bip=10.1.89.1/24 --ip-masq=true --mtu=1472"

[root@k8s-node1 ~]# cat /run/flannel/subnet.env

flannel_network=10.1.0.0/16

flannel_subnet=10.1.89.1/24

flannel_mtu=1472

flannel_ipmasq=false

确保docker启动时带有--bip=${flannel_subnet} --mtu=${flannel_mtu} 这样docker0才会成为flannel0的子网这个启动参数是通过execstartpost=/usr/local/bin/mk-docker-opts.sh -k docker_network_options -d /run/flannel/docker 实现的

-kset the combined options key to this value (default docker_opts=)

-dpath to docker env file to write to. defaults to /run/docker_opts.env

最后才启动docker

[root@k8s-node1 ~]# systemctl start docker.service

5），最后确认效果

完成后确认网络接口docker0的ip地址属于flannel0的子网

网络启动后node1和node2节点会添加很多路由条目，并且会自动开启防火墙虽然之前我们关闭了，里面有很多策略目的是node直接的docker0网络可以互通，这样各个node间通过物理网卡--flannel0--docker0和容器通信

[root@k8s-node1 ~]# ip addr

6: flannel0: mtu 1472 qdisc pfifo_fast state unknown qlen 500

link/none

inet 10.1.89.0/16 scope global flannel0

valid_lft forever preferred_lft forever

7: docker0: mtu 1500 qdisc noqueue state down

link/ether 02:42:f1:e4:7c:a3 brd ff:ff:ff:ff:ff:ff

inet 10.1.89.1/24 scope global docker0

valid_lft forever preferred_lft forever

[root@k8s-node2 ~]# ip addr

6: docker0: mtu 1500 qdisc noqueue state down

link/ether 02:42:33:a8:38:21 brd ff:ff:ff:ff:ff:ff

inet 10.1.8.1/24 scope global docker0

valid_lft forever preferred_lft forever

7: flannel0: mtu 1472 qdisc pfifo_fast state unknown qlen 500

link/none

inet 10.1.8.0/16 scope global flannel0

valid_lft forever preferred_lft forever

node1上 ping node2的docker0能通就行

[root@k8s-node1 ~]# ping 10.1.8.1

ping 10.1.8.1 (10.1.8.1) 56(84) bytes of data.

64 bytes from 10.1.8.1: icmp_seq=1 ttl=62 time=0.498 ms

64 bytes from 10.1.8.1: icmp_seq=2 ttl=62 time=0.463 ms

阅读(3399) | 评论(0) | 转发(1) |

上一篇：redis3.0.7集群部署完整版

下一篇：运维的独孤九剑

给主人留下些什么吧！~~

| | | | |

感谢所有关心和支持过chinaunix的朋友们