Container Orchestration
Kubernetes 환경 구성 방법
bong2.
2022. 10. 8. 17:37
1. 구성도
예시
이해를 돕기 위한 참고 사진이며, 실제 구축한 테스트 환경과는 차이가 있습니다.
- 실제 구축 환경
- Master Node: 1
- Worker Node: 2
- Pod per Worker: 2
2. Kubernetes 설치
2-1) Master / Worker Node 공통
사전 설치 환경 구성
2.1.1) hostname 변경
👇변경 방법
# Master node server
[root@hyunseok ~]# hostnamectl set-hostname hyunseok.master.node
[root@hyunseok ~]# hostname
hyunseok.master.node
# Worker node server
[root@hyunseok ~]# hostnamectl set-hostname hyunseok.worker.node
[root@hyunseok ~]# hostname
hyunseok.worker.node2.1.2) /etc/hosts 파일 수정
/etc/hosts 파일에 cluster를 구성할 서버의 IP와 hostname을 입력
👇vi /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.188.1 hyunseok.master.node
192.168.188.2 hyunseok.worker1.node
192.168.188.3 hyunseok.worker2.node2.1.3) selinux disabled
👇selinux 끄는 법
# selinux 상태 확인 (예: enabled 상태)
[root@hyunseok ~]# sestatus
SELinux status: enabled
SELinuxfs mount: /sys/fs/selinux
SELinux root directory: /etc/selinux
Loaded policy name: targeted
Current mode: enforcing
Mode from config file: enforcing
Policy MLS status: enabled
Policy deny_unknown status: allowed
Max kernel policy version: 28
# 끄는 방법
SELinux status: enabled -> disabled 로 변경 후 저장
[root@hyunseok ~]# reboot
[root@hyunseok ~]# sestatus (예: disabled 상태)
SELinux status: disabled2.1.4) 스왑 오프
👇swap off 방법
# 명령어 수행
[root@hyunseok ~]# swapoff -a
# /etc/fstab 파일 수정
[root@hyunseok ~]# vi /etc/fstab
#
# /etc/fstab
# Created by anaconda on Mon Nov 16 13:44:40 2020
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/centos-root / xfs defaults 0 0
UUID=281f8916-8dda-4194-8b6c-5bef589e8d99 /boot xfs defaults 0 0
/dev/mapper/centos-home /home xfs defaults 0 0
#/dev/mapper/centos-swap swap swap defaults 0 0 해당 라인 주석 처리2.1.5) 방화벽 종료
👇방화벽 끄는 법
# CentOS/RHEL의 경우
[root@hyunseok ~]# systemctl stop firewalld
[root@hyunseok ~]# systemctl disable firewalld
# Ubuntu의 경우
[root@hyunseok ~]# systemctl stop ufw
[root@hyunseok ~]# systemctl disable ufw2.1.6) Docker 설치
👇설치 방법
# CentOS/RHEL의 경우
[root@hyunseok ~]# yum install -y docker
[root@hyunseok ~]# systemctl start docker
[root@hyunseok ~]# systemctl enable docker
# Ubuntu의 경우
[root@hyunseok ~]# apt update
[root@hyunseok ~]# apt install apt-transport-https ca-certificates curl software-properties-common
[root@hyunseok ~]# curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
[root@hyunseok ~]# add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu bionic stable"
[root@hyunseok ~]# apt update
[root@hyunseok ~]# apt-cache policy docker-ce
[root@hyunseok ~]# apt install docker-ce2.1.7) kubernetes 설치
👇설치 방법
# CentOS/RHEL의 경우
[root@hyunseok ~]# vi /etc/yum.repo.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
[root@hyunseok ~]# yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes
[root@hyunseok ~]# systemctl enable --now kubelet
[root@hyunseok ~]# reboot
# Ubuntu의 경우
[root@hyunseok ~]# curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add
[root@hyunseok ~]# apt-add-repository "deb http://apt.kubernetes.io/ kubernetes-xenial main"
[root@hyunseok ~]# apt-get install kubeadm kubelet kubectl
[root@hyunseok ~]# apt-mark hold kubeadm kubelet kubectl
[root@hyunseok ~]# systemctl enable --now kubelet
[root@hyunseok ~]# reboot2-2) Master Node
- Master Node only 명령어
- --apiserver-adverties-address
- Master Node Server IP
- --pod-network-cidr
- 10.244.0.0/16
- --apiserver-adverties-address
👇 Cluber 가입 명령어
[root@hyunseok ~]# kubeadm init --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=[Master Node Server IP]위 명령어 수행 후 표기되는 "kubeadm join ~" 내용 copy
👇 kubectl 명령 수행을 위한 환경 변수 설정
[root@hyunseok ~]# export KUBECONFIG=/etc/kubernetes/admin.conf👇 CNI 설치 명령어
[root@hyunseok ~]# kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml2-3) Worker Node
- Worker only 명령어
- 위에서 copy 한 "kubeadm join ~" paste
👇 Master Node에 소속되기
[root@hyunseok ~]# kubeadm join 192.168.188.11:6443 --token 6tb1t2.o4fwnlzk4swzfr06 --discovery-token-ca-cert-hash sha256:c5f0b7432fbaaaaa1df534a18c6d36f47a833091042e2e1845a8b5adb42c56f2- 소속 여부 확인
- Master Node에서 아래 명령어 수행
👇 소속 여부 확인 명령어
[root@hyunseok ~]# kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
hyunseok.master.node Ready master 4h19m v1.19.4 192.168.188.1 <none> CentOS Linux 7 (Core) 3.10.0-1062.el7.x86_64 docker://1.13.1
hyunseok.worker1.node Ready <none> 4h16m v1.19.4 192.168.188.2 <none> CentOS Linux 7 (Core) 3.10.0-1062.el7.x86_64 docker://1.13.1
hyunseok.worker2.node Ready <none> 4h16m v1.19.4 192.168.188.3 <none> CentOS Linux 7 (Core) 3.10.0-1062.el7.x86_64 docker://1.13.13. 테스트 환경 구성 가이드
3-1) Ingress Controller
Ingress Controller 설치
👇 ingress-nginx 설치 및 확인
# ingress-nginx 설치 명령어
[root@hyunseok ~]# kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/cloud/deploy.yaml
# ingress-nginx 설치 확인 명령어 1
[root@hyunseok ~]# kubectl get namespace
NAME STATUS AGE
default Active 4d2h
ingress-nginx Active 4d2h # 해당 namespace가 생성되어야 한다.
kube-node-lease Active 4d2h
kube-public Active 4d2h
kube-system Active 4d2h
metallb-system Active 4d2h
# ingress-nginx 설치 확인 명령어 2
[root@hyunseok ~]# kubectl get all -n ingress-nginx
NAME READY STATUS RESTARTS AGE
pod/ingress-nginx-admission-create-ff429 0/1 Completed 0 4d2h
pod/ingress-nginx-admission-patch-4wt8z 0/1 Completed 1 4d2h
pod/ingress-nginx-controller-c4f944d4d-k72j9 1/1 Running 0 4d2h
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/ingress-nginx-controller LoadBalancer 10.98.18.67 <pending> 80:30349/TCP,443:30374/TCP 4d2h # ingress 설정이 완료되면 pending -> IP address 표기로 변경된다.
service/ingress-nginx-controller-admission ClusterIP 10.111.192.188 <none> 443/TCP 4d2h
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/ingress-nginx-controller 1/1 1 1 4d2h
NAME DESIRED CURRENT READY AGE
replicaset.apps/ingress-nginx-controller-c4f944d4d 1 1 1 4d2h
NAME COMPLETIONS DURATION AGE
job.batch/ingress-nginx-admission-create 1/1 21s 4d2h
job.batch/ingress-nginx-admission-patch 1/1 21s 4d2h3-2) yaml 설정
3.2.1) metallb
👇 metallb.yaml
# metallb 설치 명령어
[root@hyunseok ~]# kubectl apply -f https://raw.githubusercontent.com/google/metallb/v0.8.3/manifests/metallb.yaml
# metallb 설치 확인 명령어
[root@hyunseok kubeyaml]# kubectl get namespaces
NAME STATUS AGE
default Active 4d2h
ingress-nginx Active 4d2h
kube-node-lease Active 4d2h
kube-public Active 4d2h
kube-system Active 4d2h
metallb-system Active 4d2h # 해당 namespace가 생성되어야 한다.
# metallb.yaml 예시
[root@hyunseok ~]# cat metallb.yaml
apiVersion: v1
kind: ConfigMap
metadata:
namespace: metallb-system
name: config
data:
config: |
address-pools:
- name: default
protocol: layer2
addresses:
- 192.168.188.6-192.168.188.10
# yaml 파일 적용
[root@hyunseok ~]# kubectl apply -f metallb.yaml3.2.2) deployment
👇 deployment.yaml
# deployment.yaml 예시
[root@hyunseok ~]# cat deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx
spec:
selector:
matchLabels:
app: nginx
replicas: 4
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1
ports:
- name: http
containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: nginx
spec:
ports:
- name: http
port: 80
protocol: TCP
targetPort: 80
selector:
app: nginx
# yaml 파일 적용
[root@hyunseok ~]# kubectl apply -f deployment.yaml3.2.3) ingress
👇 ingress.yaml
# ingress.yaml 예시
[root@hyunseok ~]# cat ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: hyunseok-ingress-nginx
namespace: default
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
rules:
- host: hyunseok.centos7.com
http:
paths:
- path: /testpath
backend:
serviceName: nginx
servicePort: 80
# yaml 파일 적용
[root@hyunseok ~]# kubectl apply -f ingress.yaml3-3) Ingress 확인 방법
설정 확인
👇 ingress 확인 방법
# 확인 명령어 1
[root@hyunseok ~]# kubectl get ing
Warning: extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
NAME CLASS HOSTS ADDRESS PORTS AGE
hyunseok-ingress-nginx <none> hyunseok.centos7.com 192.168.188.6 80 4d2h # ADDRESS에 IP가 정상적으로 표기되어야 한다.
# 확인 명령어 2
[root@hyunseok ~]# kubectl get svc -n ingress-nginx
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress-nginx-controller LoadBalancer 10.98.18.67 192.168.188.6 80:30349/TCP,443:30374/TCP 4d2h # EXTERNAL-IP에 IP가 정상적으로 표기되어야 한다.
ingress-nginx-controller-admission ClusterIP 10.111.192.188 <none> 443/TCP 4d2h
# 확인 명령어 3
[root@hyunseok ~]# kubectl get all -n ingress-nginx
NAME READY STATUS RESTARTS AGE
pod/ingress-nginx-admission-create-ff429 0/1 Completed 0 4d2h
pod/ingress-nginx-admission-patch-4wt8z 0/1 Completed 1 4d2h
pod/ingress-nginx-controller-c4f944d4d-k72j9 1/1 Running 0 4d2h
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/ingress-nginx-controller LoadBalancer 10.98.18.67 192.168.188.6 80:30349/TCP,443:30374/TCP 4d2h
service/ingress-nginx-controller-admission ClusterIP 10.111.192.188 <none> 443/TCP 4d2h
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/ingress-nginx-controller 1/1 1 1 4d2h
NAME DESIRED CURRENT READY AGE
replicaset.apps/ingress-nginx-controller-c4f944d4d 1 1 1 4d2h
NAME COMPLETIONS DURATION AGE
job.batch/ingress-nginx-admission-create 1/1 21s 4d2h
job.batch/ingress-nginx-admission-patch 1/1 21s 4d2h👇 hosts 파일 변경
[root@hyunseok ~]# vi /etc/hosts
192.168.188.6 hyunseok.centos7.com # 해당 내용 추가
4. 테스트 환경 구축 완료
결과 확인
- 정상 동작 예시
- 도메인명으로 접근 시 html 내용 표기
- IP로 접근 시 404 error
