[TOC]
根据马哥教育k8s教程学习。
k8s学习大纲 - 基础大纲 - 集群部署及陈述式命令管理 - 资源类型及配置清单 - pod资源 - pod 控制器 - service资源 - 存储卷 - configmap与secret资源 - statefulset控制器 - 认证、授权及准入控制 - 网络模型及网络策略 - pod资源调度 - crd、自定义资源、自定义控制器及自定义API server, cni=Custom/Container Network Interface, crd=CustomResourceDefinition, CRI=Container Runtime Interface - 资源指标与HPA控制器 - Helm管理器 - 高可用kubernetes Cloud Native Apps: 云原生应用，程序开发出来就是运行在云平台上，而非单机上的应用。 Serverless: 与云原生应用组合，成为新的发展趋势，FaaS。 Knative。 单体应用程序: 亦称巨石应用，牵一发而动全身。 分成架构服务: 每个团队维护一个层次。（比如用户，商品，支付） 微服务(Microservice): 服务拆分成多个小功能，单独运行。 - 服务注册和服务发现: 分布式和微服务存在的问题 - 三五个程序运行支撑的服务转变为 三五十个微型服务，服务之间的调用成为网状结构。 - 非静态配置，动态服务发现，服务总线 - 服务编排系统: 解决运维部署难的问题 - 容器编排系统: 解决系统异构，服务依赖环境各不相同。 服务编排系统 --> 容器编排系统 容器编排系统: (主流: k8s, Borg, Docker Swarm, Apache Mesos Marathon DC/OS ) what is container orchestration? - container orchestration is all about managing the lifecycles of containers, especially in large, dynamic environments. - software teams use container orchestration to control and automade many task: - provisioning and deployment of containers - redundancy and availability of containers - scaling up or removing containers to spread application load evently across host infrastructure - movement of containers from one host to another if there is a shortage of resources in host, or if a host dies. - allocation of resources between containers - exteral exposure of services running in a container with the outside world - load balancing of service discovery between containers - heath monitoring of containers and hosts - configuration of an application in relation to the containers runing it 简单来说，容器编排是指容器应用的自动布局、协同及管理，它主要负责完成以下具体内容： service discovery load balancing secrets/configuration/storage management heath checks auto-[scaling/restart/healing] of containers and nodes zero-downtime deploys - pod资源 - pod 控制器: pod创建方式: - 自助式pod: 直接创建pod - 由控制器管理pod: 创建deployment,service等 ReplicationControler: ReplicaSet/rs: - 副本数 - 标签选择 - pod资源摸版 - kubectl explain rs - kubectl explain rs.spec - replicas - selector - template Deployment: 无状态任务,关注群体行为. 只关注数量,不关注个体. - kubectl explain deploy.spec - replicas - selector - template - strategy: 更新策略 - rollingUpdate - maxSurge: 最多超出目标pod数量 1/20% - maxUnavailable: 最多不可用数量. 1/80% - type - Recreate - rollingUpdate - minReadySeconds - revisionHistoryLimit: 保存历史版本数量限制 DaemonSet/ds: 每个node节点只运行一个,eg:ELK. - kubectl explain ds.spec - selector - template - minReadySeconds - revisionHistoryLimit: 保存历史版本数量限制 - updateStrategy Service: 工作模式: userspace, iptables, ipvs. (kube-proxy) userspace: 1.1- iptables: 1.10- ipvs: 1.11+ - NodePort/ClusterIP : client --> NodeIP:NodePort --> ClusterIP:ServicePort --> PodIP:containerPort - LBAAS(LoadBalancerAsAService): 公有云环境中. - LoadBalancer - ExternalName - FQDN - CNAME -> FQDN - No ClusterIP: Headless Service - ServiceName -> PodIP - kubectl explain svc - type: ExternalName, ClusterIP, NodePort, LoadBalancer - port: - NodePort # type=NodePort 时 可用. client --> NodeIP:NodePort --> ClusterIP:ServicePort --> PodIP:containerPort - LBAAS(LoadBalancerAsAService): 公有云环境中. - LoadBalancer - ExternalName - FQDN - CNAME -> FQDN - - port - targetPort Ingress: Service: ingress-nginx (NodePort, DaemonSet - HostNetwork) IngressController: ingress-nginx Ingress: - site1.ikubernetes.io (virtual host) - site2.ikubernetes.io (virtual host) - example.com/path1 - example.com/path2 - Service: site1 - pod1 - pod2 - Service: site2 - pod3 - pod4 - 存储卷 - emptyDir - 临时目录, 内存使用, 没有持久性 - gitRepo - hostPath - 共享存储 - SAN: iSCSI - NAS: nfs, cifs, http - 分布式存储: - glusterfs - ceph: rbd - cephfs: - 云存储 - EBS - Azure Disk - pvc: persistentVolumeClaim - pv - pvc - pod - volumes - - secret base64 的 configmap - configmap: 配置中心 配置容器化应用的方式: 1. 自定义命令行参数; args: [] 2.把配置文件直接copy进镜像 3.环境变量 1.Cloud Native的应用程序一般可直接通过环境变量加载配置 2.通过entrypoint.sh脚本来预处理变量为配置文件中的配置信息 4.存储卷 - 变量注入 --> pod 读取变量 - 挂载存储卷 --> pod 读取配置 - 自定义命令行参数 - StatefulSet: PetSet -> StatefulSet 应对有以下要求的服务: 1.稳定且唯一的网络标识符 2.稳定且持久的存储 3.有序平滑的部署和扩展 4.有序平滑的删除和终止 5.有序的滚动更新 三个组件: - headless service: clusterIP: None - StatefulSet - volumeClaimTemplate - 认证、授权及准入控制 - Authentication - restful API: token - tls: 双向认证 - user/password - Authorization - rbac - role - rolebinding - ClusterRole - ClusterRoleBinding - webhook - abac - Admission Control client --> API Server pod --> api server - ServiceAccount? - secret - ServiceAccountName kubectl get secret kubectl get sa | serviceaccount kubectl create serviceaccount my-sa -o yaml --dry-run kubectl get pods myapp -o yaml --export API request path: http://IP:port/apis/apps/v1/namespaces/default/deployments/myapp-deploy/ http request ver: get, post, put, delete http request --> API requests verb: get, list, create, update, patch, watch, proxy, redirect, delete, delectcollection Resource: Subresource: Namespace: API group: - ServiceAccount: alex.crt (umask 077; openssl genrsa -out alex.key 2048) openssl req -new -key alex.key -out alex.csr -subj "/CN=alex" openssl x509 -req -in alex.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out alex.crt -days 3650 openssl x509 -in alex.crt -text -noout kubectl config set-cluster alex-cluster --server="https://192.168.137.131:6443" --certificate-authority=/etc/kubernetes/pki/ca.crt --embed-certs=true --kubeconfig=/tmp/alex.conf kubectl config set-credentials admin \ --client-certificate=datasets/work/admin/admin.pem \ --client-key=datasets/work/admin/admin-key.pem \ --embed-certs=true \ --kubeconfig=/tmp/alex.conf kubectl config set-context alex@kubernetes --cluster=kubernetes --user=admin --kubeconfig=/tmp/alex.conf kubectl config use-context kubernetes --kubeconfig=/tmp/alex.conf kubectl get pods --kubeconfig=/tmp/alex.conf (error from server forbidden) - RBAC - role, clusterrole object: resource group resource non-resource url action: get, list, watch, patch, delete, deletecollection, ... - rolebinding, clusterrolebinding subject: user group serviceaccount role: - role: - operations - objects - rolebinding: - user account or service account - role - clusterrole - clusterrolebinding # kubectl create role --help # kubectl create rolebinding --help # kubectl create role pods-reader --verb=get,list,watch --resource=pods --dry-run=client -o yaml apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: creationTimestamp: null name: pods-reader namespace: default rules: - apiGroups: - "" resources: - pods verbs: - get - list - watch # kubectl create rolebinding alex-read-pods --role=pods-reader --user=alex --dry-run=client -o yaml apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: creationTimestamp: null name: alex-read-pods roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: pods-reader subjects: - apiGroup: rbac.authorization.k8s.io kind: User name: alex # 接着上面 ServiceAccount 的例子 # kubectl config use-context kubernetes --kubeconfig=/tmp/alex.conf # kubectl get pods --kubeconfig=/tmp/alex.conf # kubectl get pods -n kube-system --kubeconfig=/tmp/alex.conf (error from server forbidden) # kubectl create clusterrole --help # kubectl create clusterrolebinding --help - Kubernetes Dashboard - helm install kubernetes-dashboard/kubernetes-dashboard --version 2.3.0 --name=k8s-dashboard [--namespaces=dashboard] - helm fetch kubernetes-dashboard/kubernetes-dashboard --version 2.3.0 - tar xf kubernetes-dashboard-2.3.0.tgz - vim kubernetes-dashboard/values.yaml - helm install kubernetes-dashboard/kubernetes-dashboard --version 2.3.0 --name=k8s-dashboard -f kubernetes-dashboard/values.yaml [--namespaces=dashboard] - kubectl create clusterrolebinding k8s-dashboard-admin --clusterrole=cluster-admin --serviceaccount=default:k8s-dashboard / [--serviceaccount=dashboard:k8s-dashboard] - kubectl describe sa k8s-dashboard [-n dashboard] - kubectl describe secret k8s-dashboard-token-xxxx [-n dashboard] - 网络模型及网络策略 flannel kubectl get daemonset -n kube-system kubectl get pods -o wide -n kube-system |grep -i kube-flannel kubectl get configmap -n kube-system kubectl get configmap kube-flannel-cfg -o json -n kube-system from 10.244.1.59 ping 10.244.2.76 tcpdump -i cni0 -nn icmp tcpdump -i flannel.1 -nn tcpdump -i ens32 -nn host 192.168.137.131 overlay otv calico - pod资源调度 调度器: 预选策略: 优先函数: 节点选择器: nodeSelector, nodeName 节点亲和调度: nodeAffinity taint的effect定义对Pod排斥效果: NoSchedule:仅影响调度过程,对现存的Pod对象不产生影响; NoExecute:既影响调度过程,也影响现在的Pod对象;不容忍的Pod对象将被驱逐; PreferNoSchedule: - crd、自定义资源、自定义控制器及自定义API server - HeapSter (数据采集) - cAdvisor (数据指标检测) - InfluxDB (历史数据记录: 时序数据库系统) - Grafana (数据展示) - RBAC 资源指标: metrics-server: k8s资源聚合器 自定义指标: - prometheus - k8s-prometheus-adapter - MertricServer - PrometheusOperator - NodeExporter - kubeStateMetrics - Prometheus - Grafana 新一代架构: 核心指标流水线: 由kubelet、metrics-server以及由API server提供的api组成;CPU累积使用率、内存实时使用率、Pod的资源占用率及容器的磁盘占用率; 监控流水线: 用于从系统收集各种指标数据并提供终端用户、存储系统以及HPA,它们包含核心指标及许多非核心指标。非核心指标本身不能被k8s所解析, metrics-server: API server - Helm管理器: chart repository Tiller: chart: - 配置清单 - 模板文件 - helm install mem1 stable/memcached # kubectl explain ingress.spec FIELDS: backend <Object> resource <Object> serviceName <string> servicePort <string> ingressClassName <string> rules <[]Object> host <string> http <Object> paths <[]Object> -required- backend <Object> -required- path <string> pathType <string> tls <[]Object> Ingress Controller: - Nginx - Traefik - Envoy namespace: ingress-nginx Job: 一次性任务 Cronjob: 周期性任务 StatefulSet: 关注个体. 对 EDR: Custom Defined Resources, 1.8+ Operator: etcd example: - # ReplicaSet - vim rs-demo.yaml apiVersion: apps/v1 kind: ReplicaSet metadata: name: myapp namespace: default spec: replicas: 2 selector: matchLabels: app: myapp release: canary template: metadata: name: myapp-pod labels: app: myapp release: canary environments: qa sepe: containers: - name: myapp-container image: ikubernetes/myapp:v1 ports: - name: http containerPort: 80 - kubectl create -f rs-demo.yaml - kubectl edit rs myapp --> replicas: 5 - kubectl get pods - kubectl edit rs myapp --> image: ikubernetes/myapp:v2 - kubectl get rs -o wide - curl xx.xx.xxx.xx # 只有重建的pod会使用新的image - # Deployment - vim deploy-demo.yaml apiVersion: apps/v1 kind: Deployment metadata: name: myapp-deploy namespace: default spec: replicas: 2 selector: matchLabels: app: myapp release: canary template: metadata: name: myapp-pod labels: app: myapp release: canary environments: qa sepe: containers: - name: myapp-container image: ikubernetes/myapp:v1 ports: - name: http containerPort: 80 - kubectl apply -f deploy-demo.yaml - kubectl get deploy - kubectl get rs -o wide myapp-deploy-69b47bc96d --> 模板的hash值 - kubectl get pods - kubectl get pods -l app=myapp -w # 新开窗口 - vim deploy-demo.yaml --> image: ikubernetes/myapp:v2 - kubectl apply -f deploy-demo.yaml - kubectl get rs -o wide - kubectl rollout history deployment myapp-deploy # 打补丁方式增加pod - kubectl patch deployment myapp-deploy -p '{"spec":{"replicas":5}}' - kubectl get pods - kubectl patch deployment myapp-deploy -p '{"spec":{"strategy":{"rollingUpdate":{"maxSurge":1,"maxUnavailable":0}}}}' - kubectl describe deployment myapp-deploy - kubectl get pods -l app=myapp -w # 新开窗口-1 - kubectl set image deployment myapp-deploy myapp=ikubernetes/myapp:v3 && kubectl rollout pause deployment myapp-deploy # 暂停rollout, 金丝雀发布 - kubectl rollout status deployment myapp-deploy # 新开窗口-2 - kubectl rollout resume deployment myapp-deploy --> # 查看新开窗口 1, 2 - kubectl rollout history deployment myapp-deploy # 查看历史版本 - kubectl get rs -o wide # 回滚版本 - kubectl rollout undo --help - kubectl rollout undo deployment myapp-deploy --to-revision=1 - kubectl rollout history deployment myapp-deploy # 查看历史版本 - # DaemonSet - vim ds-demo.yaml apiVersion: apps/v1 kind: Deployment metadata: name: redis namespaces: default spec: replicas: 1 selector: matchLabels: app: redis role: logstor template: metadata: labels: app: redis role: logstor spec: containers: - name: redis image: redis:4.0-alpine ports: - name: redis containerPort: 6379 --- apiVersion: apps/v1 kind: DaemonSet metadata: name: filebeat-ds namespace: default spec: replicas: 2 selector: matchLabels: app: filebeat release: stable template: metadata: name: filebeat-pod labels: app: filebeat release: stable sepe: containers: - name: filebeat image: ikubernetes/filebeat:5.6.5-alphine env: - name: REDIS_HOST value: redis.default.svc.cluster.local - name: REDIS_LOG_LEVEL value: info - kubectl apply -f ds-demo.yaml - kubectl get pods - kubectl expose deployment redis --port=6379 - kubect get svc - kubectl exec -it redist-5bxxxx-xxx -- /bin/sh # netstat -tnl # nslookup redis.default.svc.cluster.local # redis-cli -h redis.default.svc.cluster.local # kyes * # DaemonSet 滚动更新 - kubectl set image demonsets filebeat-ds filebeat=ikubernets/filebeat-5.6.6-alphine - kubectl get pods -w # 一个一个的更新. - # Service - vim redis-svc.yaml apiVersion: v1 kind: Service medatada: name: redis namespace: default spec: selector: app: redis role: logstor # clusterIP: 10.96.96.96 type: ClusterIP port: - port: 6349 targetPort: 6379 - kubectl apply -f redis-svc.yaml - kubectl get svc - kubectl describe svc redis - - curl xx.xx.xxx.xx # 只有重建的pod会使用新的image - service资源 Helm: like Linux yum/apt/apk ... k8s Server Master API Server - port: 6443 - auth: 双向认证, /etc/kubernetes/pki - config: ~/.kube/config - kubectl config view - restful api - kubectl api-versions - json - kubectl get pod == curl https://master:6443/v1/... - communicate with all the service - kubectl - kube-controller-manager - kube-scheduler - etcd - kubelet - kube-proxy - api 接口中的资源分成多个逻辑组合 apiVersion - 和解循环（Reconciliation Loop）: status --> spec Scheduler Controller - 和解循环（Reconciliation Loop）: status --> spec Node: - pod - service -> iptables/ipvs -> kube-proxy - kube-proxy Resource 资源有两个级别： - 集群级别 - Node - Namespace - Role - ClusterRole - RoleBinding - ClusterRoleBinding - PersistentVolume - 名称空间级别 - pod - service - deploy - 元数据型资源 - HPA - PodTemplate - LimitRange 资源组成部分： - apiVersion: group/version (kubectl api-versions) - kind: 资源类别 - metadata - spec: 资源期望状态 - labels/tag: kubectl label - annotations: kubectl annotate - initContainers: kubectl explain pods.spec.initContainers: 容器初始化, 运行前, 运行后, 运行时 (存活检测, 就绪检测) - lifecycle - livenessProbe - readinessProbe - startupProbe - - Containers: kubectl explain pods.spec.Containers: 容器初始化, 运行前, 运行后, 运行时 (存活检测, 就绪检测) - lifecycle: preStart hook, preStop hook - livenessProbe - readinessProbe - startupProbe - name: - command: ["/bin/bash" "-c" "sleep 3600"] - args: - image: - imagePullPolicy: - Never - Always - IfNotPresent - port: - name: - hostIP: - hostPort: - protocol - containerPort - status: 资源当前状态 资源引用Object URL: /apis/<GROUP>/<VERSION>/namespaces/<NAMESPACE_NAME>/<KIND>[/OBJECT_ID]/ - /api/GROUP/VERSION/namespaces/NAMESPACE/TYPE/NAME - kubectl get pod/nginx-ds-s4hpn - selfLink: /api/v1/namespaces/default/pods/nginx-ds-s4hpn 资源记录: SVC_NAME.NS_NAME.DOMAIN.LTD. redis.default.svc.cluster.local - Pod - Pod Controller - Deployment: 类型 --> ngx-deploy --> nginx pod - Service - nginx-svc --> 关联到 nginx pod kubeadm kubectl - kubectl explain pods.spec.initContainers - kubectl -h - basic commands beginner - create - expose - run - set - basic commands intermediate - explan - get - edit - delete - deploy commands - rollout - scale - autoscale - cluster management commands - certificate - cluster-info - top - cordon - uncordon - drain - taint - troubleshooting and debugging commands - describe - logs - attach - exec: kubectl exec -it PodName -c ContainerName -- /bin/sh - port-forward - kubectl config view [-o wide/json/yaml] # kubectl config view apiVersion: v1 clusters: - cluster: certificate-authority-data: DATA+OMITTED server: https://192.168.137.131:6443 name: kubernetes contexts: - context: cluster: kubernetes user: kubernetes-admin name: kubernetes-admin@kubernetes current-context: kubernetes-admin@kubernetes kind: Config preferences: {} users: - name: kubernetes-admin user: client-certificate-data: REDACTED client-key-data: REDACTED - kubectl api-resources: 支持的资源类型 以及缩写 - kubectl get [-o wide/json/yaml] [-n default/kube-system/...] [-L label-name] [-l label-name ==/!= label-value] - all - nodes - pods - ns/namespace - default - kube-public - kube-system - deploy - svc/service - kubectl create [ -f file] - job - namespace - kubectl create namespace testing - kubectl create namespace prod - kubectl create namespace develop - kubectl get ns - kubectl delete namespace testing - kubectl delete ns/prod ns/develop - deployment - kubectl create deployment nginx-deploy --image=nginx:1.14-alpine - kubectl get all -o wide - pod/nginx-deploy-xxxx (ip: 10.244.1.2) - deployment.apps/nginx-deploy - replicaset.apps/nginx-deploy-xxxx - curl 10.244.1.2 ( welcome nginx!) - kubectl delete pod/nginx-deploy-xxxx - kubectl get all -o wide - pod/nginx-deploy-xxxx (ip: 10.244.3.2) ... - curl 10.244.3.2 ( welcome nginx!) - service - kubectl create service -h - ClusterIP - NodePort - kubectl create service clusterip nginx-deploy --tcp=80:80 (名字和上面deploy保持一致，就会自动分配IP) - kubectl get svc/nginx-deploy -o yaml - clusterip： 10.110.129.64 - endpoints: 10.244.3.2 - kubectl describe svc/nginx-deploy # 删除 pod 测试 - curl 10.110.129.64 ( welcome nginx!) - kubectl delete pod/nginx-deploy-xxxx - kubectl get all -o wide - pod/nginx-deploy-xxxx (ip: 10.244.1.3) ... - kubectl get svc/nginx-deploy -o yaml - clusterip： 10.110.129.64 - endpoints: 10.244.1.3 (自动关联到最新的pod) # 删除 service 测试 - curl nginx-deploy.default.svc.cluster.local. ( welcome nginx!) - kubectl delete svc/nginx-deploy - kubectl create service clusterip nginx-deploy --tcp=80:80 (名字和上面deploy保持一致，就会自动分配IP) - kubectl describe svc/nginx-deploy -o yaml - clusterip： 10.111.215.249 - endpoints: 10.244.3.2 - curl nginx-deploy.default.svc.cluster.local. ( welcome nginx!) # 按需伸缩 pod 测试 - kubectl create deploy myapp --image=ikubernetes/myapp:v1 - kubectl get deploy - kubectl get pods -o wide - ip : 10.244.3.3 - curl 10.244.3.3 - curl 10.244.3.3/hostname.html (show the pod name: myapp-xxxx-yyyy) - kubectl create service clusterip myapp --tcp=80:80 - kubectl describe svc/myapp - IP 10.100.182.218 - Endpoints: 10.244.3.3 - curl nginx-deploy.default.svc.cluster.local. ( welcome myapp!) - curl nginx-deploy.default.svc.cluster.local/hostname.html (show the pod name: myapp-xxxx-yyyy) - kubectl scale --replicas=3 myapp (deploy name) - kubectl describe svc/myapp - IP 10.100.182.218 - Endpoints: 10.244.3.3, 10.244.1.4, 10.244.2.2 - curl nginx-deploy.default.svc.cluster.local/hostname.html (随机显示不同IP的pod name，多次重复执行查看效果) - kubectl scale --replicas=2 myapp - kubectl describe svc/myapp - IP 10.100.182.218 - Endpoints: 10.244.3.3, 10.244.1.4 # nodeport 外网访问 - kubectl delete svc/myapp - kubectl create service nodeport -h - kubectl create service nodeport myapp --tcp=80:80 - kubectl get svc - ports: 80:31996/TCP - 集群外部访问所有nodes的 http://nodesip:31996/hostname.html - 自动创建规则在每一个节点的iptables --> kube-proxy - ssh nodes "iptables -t nat -vnL" eg: - kubectl expose - kubectl set image deployment myapp myapp=ikubernetes/myapp:v2 - kubectl label [--overwrite] (-f FILENAME | TYPE NAME) KEY_1=VAL_1 ... KEY_N=VAL_N [--resource-version=version] kubectl label pods -n dev-namespaces apptag=my-app release=stable deltag- - kubectl api-versions - kubectl describe Network - node network - service network: service 注册/发现 - pod network 外网访问: - Service: NodePort - hostport: - hostNetwork: ipvs/iptables 4 层调度器 ingress 7 层调度器