以下文章来源于k8s实战 ,作者mark
布道云原生,掌握云原生生态
【导读】本文非常详细地介绍了istio做tracing的多控制面、多网格实操步骤。
这里的可观察性主要指服务网格的可观察性,也就是需要观测服务网格中运行的微服务。为什么可观察性很重要,因为随着微服务架构的流行,一个系统可能运行成百上千微服务,如果系统出现故障,定位问题带来很大的问题。有了观测系统,就能更好的分析问题发生的原因,已经更好的监控告警。
服务网格可观察性主要分为三个大类,分别是 log,metrics,tracing。
log 是指将分布式系统的日志收集起来集中存储,用于日志分析,常用的工具如 efk。
metrics 是指收集服务网格的监控指标,进行监控告警,常用工具比如 prometheus。
tracing 是指分布式链路追踪,用于可视化显示服务调用的依赖关系,及获取延迟数据,常用工具如 zipkin,jaeger 等。
本文关注的是 tracing,这里我们用到了 zipkin 作为 tracing 工具。
在分布式系统,尤其是微服务系统中,一次外部请求往往需要内部多个模块,多个中间件,多台机器的相互调用才能完成。在这一系列的调用中,可能有些是串行的,而有些是并行的。在这种情况下,我们如何才能确定这整个请求调用了哪些应用?哪些模块?哪些节点?以及它们的先后顺序和各部分的性能如何呢?
链路追踪是分布式系统下的一个概念,它的目的就是要解决上面所提出的问题,也就是将一次分布式请求还原成调用链路,将一次分布式请求的调用情况集中展示,比如,各个服务节点上的耗时、请求具体到达哪台机器上、每个服务节点的请求状态等等。
我们这里要演示的链路追踪不是单个 istio 集群的,而是多个 istio 集群的。
我们把多个 istio 集群部署成一个联邦的 istio 集群,把多个集群的 tracing 数据在 zipkin 集中存储分析。单个 istio 集群的链路追踪相对比较简单,只需配置 istio 的 comfigmap 就行,多个集群考虑到集群的部署方式有很多,需要所有 proxy 将信息传送到统一的一个 zipkin,相对来说复杂一点。
这里我们展示两集群 istio 联邦和三集群 istio 联邦,一共 14 个案例。
两集群部署用的机子是:
cluster1
192.168.229.128 master
192.168.229.129 master
192.168.229.130 node
cluster2
192.168.229.131 master
192.168.229.132 master
192.168.229.133 node
三集群部署用的机子是;
cluster1
192.168.229.137 master
192.168.229.138 master
192.168.229.139 node
cluster2
192.168.229.140 master
192.168.229.141 master
192.168.229.142 node
cluster3
192.168.229.143 master
192.168.229.144 master
192.168.229.145 node
k8s 版本
[root@node01 ~]# kubectl version --short
Client Version: v1.21.0
Server Version: v1.21.0
istio 版本
[root@node01 ~]# istioctl version
client version: 1.11.2
control plane version: 1.11.2
data plane version: none
首先需要创建 root-ca,多个 istio 集群的 root-ca 必须是一样的:
cluster1:
mkdir -p certs
make -f ../tools/certs/Makefile.selfsigned.mk root-ca
make -f ../tools/certs/Makefile.selfsigned.mk cluster1-cacerts
make -f ../tools/certs/Makefile.selfsigned.mk cluster2-cacerts
scp -r cluster2 root@192.168.229.131:/root/cluster2
kubectl create namespace istio-system
kubectl create secret generic cacerts -n istio-system \
--from-file=cluster1/ca-cert.pem \
--from-file=cluster1/ca-key.pem \
--from-file=cluster1/root-cert.pem \
--from-file=cluster1/cert-chain.pem
cluster2:
kubectl create namespace istio-system
kubectl create secret generic cacerts -n istio-system \
--from-file=cluster2/ca-cert.pem \
--from-file=cluster2/ca-key.pem \
--from-file=cluster2/root-cert.pem \
--from-file=cluster2/cert-chain.pem
部署步骤:
集群 1
128,129,130
集群 2
131,132,133
128。129.130
route add -net 172.21.1.0 netmask 255.255.255.0 gw 192.168.229.131
route add -net 172.21.2.0 netmask 255.255.255.0 gw 192.168.229.133
route add -net 172.21.0.0 netmask 255.255.255.0 gw 192.168.229.132
route add -net 10.69.0.0 netmask 255.255.0.0 gw 192.168.229.131
131,132,133
route add -net 172.20.0.0 netmask 255.255.255.0 gw 192.168.229.128
route add -net 172.20.1.0 netmask 255.255.255.0 gw 192.168.229.129
route add -net 172.20.2.0 netmask 255.255.255.0 gw 192.168.229.130
route add -net 10.68.0.0 netmask 255.255.0.0 gw 192.168.229.128
cat <<EOF > cluster1.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
profile: demo
values:
global:
meshID: mesh1
multiCluster:
clusterName: cluster1
network: network1
meshConfig:
accessLogFile: /dev/stdout
enableTracing: true
components:
egressGateways:
- name: istio-egressgateway
enabled: true
EOF
这里我设置的 cluster1 东西向网关的 ip 试 192.168.229.100 如果用的是 loadblance,可以用下面命令获取
# export DISCOVERY_ADDRESS=$(kubectl -n istio-system get svc istio-eastwestgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
然后替换 remotePilotAddress
cat <<EOF > cluster2.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
profile: demo
values:
global:
meshID: mesh1
multiCluster:
clusterName: cluster2
network: network1
remotePilotAddress: 192.168.229.100
meshConfig:
accessLogFile: /dev/stdout
enableTracing: true
components:
egressGateways:
- name: istio-egressgateway
enabled: true
EOF
scp cluster2.yaml root@192.168.229.131:/root
istioctl install -f cluster1.yaml
/root/istio-1.11.2/samples/multicluster/gen-eastwest-gateway.sh --mesh mesh1 --cluster cluster1 --network network1 | istioctl install -y -f -
kubectl patch svc -n istio-system istio-eastwestgateway -p '{"spec":{"externalIPs":["192.168.229.100"]}}'
kubectl apply -n istio-system -f /root/istio-1.11.2/samples/multicluster/expose-istiod.yaml
cluster2 生成访问 apiserver secret
istioctl x create-remote-secret --name=cluster2 --server=https://192.168.229.131:6443 > remote-secret-cluster2.yaml
传输 secret 到 cluster1
scp remote-secret-cluster2.yaml root@192.168.229.128:/root
cluster1 应用 secret
kubectl apply -f remote-secret-cluster2.yaml
cluster2 安装 cluster2
istioctl install -f cluster2.yaml
cluster1 重启 pod
kubectl rollout restart deploy -n istio
kubectl rollout restart deploy -n istio-system
cluster2 重启 pod
kubectl rollout restart deploy -n istio
kubectl rollout restart deploy -n istio-system
cluster1 部署 zipkin
kubectl apply -f extras/zipkin.yaml -n istio-system
cluster1 增加东西向网关端口 kubectl edit svc -n istio-system istio-eastwestgateway
- name: http-zipkin
nodePort: 32197
port: 15018
protocol: TCP
targetPort: 15018
cluster1:
暴露 zipkin
visilazation/zipkin-gw-vs.yaml
kubectl apply -f zipkin-gw-vs.yaml -n istio-system
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: zipkin-gateway
spec:
selector:
istio: eastwestgateway
servers:
- port:
name: http-zipkin
number: 15018
protocol: http
hosts:
- "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: zipkin-vs
spec:
hosts:
- "*"
gateways:
- zipkin-gateway
http:
- route:
- destination:
host: zipkin.istio-system.svc.cluster.local
port:
number: 9411
cluster1,cluster2: cm istio
[root@node01 ~]# kubectl get cm istio -n istio-system -o yaml
apiVersion: v1
data:
mesh: |-
accessLogFile: /dev/stdout
enableTracing: true
defaultConfig:
discoveryAddress: istiod.istio-system.svc:15012
meshId: mesh1
proxyMetadata: {}
tracing:
sampling: 100
zipkin:
address: 192.168.229.100:15018
enablePrometheusMerge: true
enableTracing: true
rootNamespace: istio-system
trustDomain: cluster.local
meshNetworks: 'networks: {}'
暴露服务:
kubectl port-forward --address 0.0.0.0 -n istio-system zipkin-6b8c6bdc56-m2b4f 9411:9411
清理:
cluster1:
kubectl delete vs istiod-vs -n istio-system
kubectl delete gw istiod-gateway -n istio-system
kubectl delete secret istio-remote-secret-cluster2 -n istio-system
kubectl delete gw zipkin-gateway -n istio-system
kubectl delete vs zipkin-vs -n istio-system
istioctl x uninstall -f cluster1.yaml
reboot
cluster2:
istioctl x uninstall -f cluster2.yaml
reboot
集群1
128,129,130
集群2
131,132,133
给istio-system namespace 打标签 cluster1:
kubectl label namespace istio-system topology.istio.io/network=network1
cluster2:
kubectl label namespace istio-system topology.istio.io/network=network2
生成istio operator部署文件
cat <<EOF > cluster1.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
profile: demo
values:
global:
meshID: mesh1
multiCluster:
clusterName: cluster1
network: network1
meshConfig:
accessLogFile: /dev/stdout
enableTracing: true
components:
egressGateways:
- name: istio-egressgateway
enabled: true
EOF
这里我设置的cluster1东西向网关的ip试192.168.229.100 如果用的是loadblance,可以用下面命令获取
# export DISCOVERY_ADDRESS=$(kubectl -n istio-system get svc istio-eastwestgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
然后替换remotePilotAddress
cat <<EOF > cluster2.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
profile: demo
values:
global:
meshID: mesh1
multiCluster:
clusterName: cluster2
network: network2
remotePilotAddress: 192.168.229.100
meshConfig:
accessLogFile: /dev/stdout
enableTracing: true
components:
egressGateways:
- name: istio-egressgateway
enabled: true
EOF
传输部署文件到另一个集群
scp cluster2.yaml root@192.168.229.131:/root
安装istio
istioctl install -f cluster1.yaml
安装东西向网关
/root/istio-1.11.2/samples/multicluster/gen-eastwest-gateway.sh --mesh mesh1 --cluster cluster1 --network network1 | istioctl install -y -f -
配置东西向网关ip
kubectl patch svc -n istio-system istio-eastwestgateway -p '{"spec":{"externalIPs":["192.168.229.100"]}}'
暴露istiod
kubectl apply -n istio-system -f /root/istio-1.11.2/samples/multicluster/expose-istiod.yaml
暴露服务
kubectl apply -n istio-system -f /root/istio-1.11.2/samples/multicluster/expose-services.yaml
生成istiod访问apiserver secret
istioctl x create-remote-secret --name=cluster2 --server=https://192.168.229.131:6443 > remote-secret-cluster2.yaml
传输secret到cluster1
scp remote-secret-cluster2.yaml root@192.168.229.128:/root
cluster1 安装secret
kubectl apply -f remote-secret-cluster2.yaml -n istio-system
部署cluster2
istioctl install -f cluster2.yaml
生成东西向网关
/root/istio-1.11.2/samples/multicluster/gen-eastwest-gateway.sh --mesh mesh1 --cluster cluster2 --network network2 | istioctl install -y -f -
配置东西向网关ip
kubectl patch svc -n istio-system istio-eastwestgateway -p '{"spec":{"externalIPs":["192.168.229.101"]}}'
暴露服务
kubectl apply -n istio-system -f /root/istio-1.11.2/samples/multicluster/expose-services.yaml
重启pod
kubectl rollout restart deploy -n istio
kubectl rollout restart deploy -n istio-system
cluster1 重启pod
kubectl rollout restart deploy -n istio
kubectl rollout restart deploy -n istio-system
cluster1 部署 zipkin
kubectl apply -f extras/zipkin.yaml -n istio-system
cluster1 增加东西向网关端口
kubectl edit svc -n istio-system istio-eastwestgateway - name: http-zipkin nodePort: 32197 port: 15018 protocol: TCP targetPort: 15018
cluster1 暴露 zipkin visilazation/zipkin-gw-vs.yaml
kubectl apply -f zipkin-gw-vs.yaml -n istio-system
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: zipkin-gateway
spec:
selector:
istio: eastwestgateway
servers:
- port:
name: http-zipkin
number: 15018
protocol: http
hosts:
- "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: zipkin-vs
spec:
hosts:
- "*"
gateways:
- zipkin-gateway
http:
- route:
- destination:
host: zipkin.istio-system.svc.cluster.local
port:
number: 9411
cluster1,cluster2 : cm istio
cluster1,cluster2,cluster3: cm istio
apiVersion: v1
data:
mesh: |-
accessLogFile: /dev/stdout
enableTracing: true
defaultConfig:
discoveryAddress: istiod.istio-system.svc:15012
meshId: mesh1
proxyMetadata: {}
tracing:
sampling: 100
zipkin:
address: 192.168.229.100:15018
enablePrometheusMerge: true
enableTracing: true
rootNamespace: istio-system
trustDomain: cluster.local
meshNetworks: 'networks: {}'
修改
sampling: 100
zipkin:
address: 192.168.229.100:15018
暴露服务:
kubectl port-forward --address 0.0.0.0 -n istio-system zipkin-6b8c6bdc56-m2b4f 9411:9411
清理:
cluster1:kubectl label namespace istio-system topology.istio.io/network-kubectl delete vs istiod-vs -n istio-systemkubectl delete gw istiod-gateway -n istio-systemkubectl delete gw cross-network-gateway -n istio-systemkubectl delete secret istio-remote-secret-cluster2 -n istio-systemkubectl delete gw zipkin-gateway -n istio-systemkubectl delete vs zipkin-vs -n istio-systemistioctl x uninstall -f cluster1.yamlrebootcluster2:kubectl label namespace istio-system topology.istio.io/network-kubectl delete gw cross-network-gateway -n istio-systemistioctl x uninstall -f cluster2.yamlreboot
两集群网络联通
集群1
128,129,130
集群2
131,132,133
#两个网络联通
128。129.130
route add -net 172.21.1.0 netmask 255.255.255.0 gw 192.168.229.131
route add -net 172.21.2.0 netmask 255.255.255.0 gw 192.168.229.133
route add -net 172.21.0.0 netmask 255.255.255.0 gw 192.168.229.132
route add -net 10.69.0.0 netmask 255.255.0.0 gw 192.168.229.131
131,132,133
route add -net 172.20.0.0 netmask 255.255.255.0 gw 192.168.229.128
route add -net 172.20.1.0 netmask 255.255.255.0 gw 192.168.229.129
route add -net 172.20.2.0 netmask 255.255.255.0 gw 192.168.229.130
route add -net 10.68.0.0 netmask 255.255.0.0 gw 192.168.229.128
生成部署operator文件
cat <<EOF > cluster1.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
profile: demo
values:
global:
meshID: mesh1
multiCluster:
clusterName: cluster1
network: network1
meshConfig:
accessLogFile: /dev/stdout
enableTracing: true
components:
egressGateways:
- name: istio-egressgateway
enabled: true
EOF
这里我设置的cluster1东西向网关的ip试192.168.229.100 如果用的是loadblance,可以用下面命令获取
# export DISCOVERY_ADDRESS=$(kubectl -n istio-system get svc istio-eastwestgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
然后替换remotePilotAddress
cat <<EOF > cluster2.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
profile: demo
values:
global:
meshID: mesh1
multiCluster:
clusterName: cluster2
network: network1
remotePilotAddress: 192.168.229.100
meshConfig:
accessLogFile: /dev/stdout
enableTracing: true
components:
egressGateways:
- name: istio-egressgateway
enabled: true
EOF
传输部署文件到另一个集群
scp cluster2.yaml root@192.168.229.131:/root
安装cluster1
istioctl install -f cluster1.yaml
生成东西向网关
/root/istio-1.11.2/samples/multicluster/gen-eastwest-gateway.sh --mesh mesh1 --cluster cluster1 --network network1 | istioctl install -y -f -
配置东西向网关ip
kubectl patch svc -n istio-system istio-eastwestgateway -p '{"spec":{"externalIPs":["192.168.229.100"]}}'
暴露istiod
kubectl apply -n istio-system -f /root/istio-1.11.2/samples/multicluster/expose-istiod.yaml
cluster2 生成访问apiserver secret
istioctl x create-remote-secret --name=cluster2 --server=https://192.168.229.131:6443 > remote-secret-cluster2.yaml
传输secret到cluster1
scp remote-secret-cluster2.yaml root@192.168.229.128:/root
cluster1 应用secret
kubectl apply -f remote-secret-cluster2.yaml
cluster2 安装cluster2
istioctl install -f cluster2.yaml
cluster1 重启pod
kubectl rollout restart deploy -n istio
kubectl rollout restart deploy -n istio-system
cluster2 重启pod
kubectl rollout restart deploy -n istio
kubectl rollout restart deploy -n istio-system
部署 zipkin
kubectl apply -f extras/zipkin.yaml -n istio-system
由于 cluster2 dns 无法解析 zipkin.istio-system,所以 cluster1 需要安装东西向网关
# 部署东西向网关
/root/istio-1.11.2/samples/multicluster/gen-eastwest-gateway.sh --mesh mesh1 --cluster cluster1 --network network1 | istioctl install -y -f -
#配置东西向网关ip
kubectl patch svc -n istio-system istio-eastwestgateway -p '{"spec":{"externalIPs":["192.168.229.100"]}}'
cluster1 增加东西向网关端口
kubectl edit svc -n istio-system istio-eastwestgateway
- name: http-zipkin
nodePort: 32197
port: 15018
protocol: TCP
targetPort: 15018
cluster1:
暴露 zipkin
visilazation/zipkin-gw-vs.yaml
kubectl apply -f zipkin-gw-vs.yaml -n istio-system
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: zipkin-gateway
spec:
selector:
istio: eastwestgateway
servers:
- port:
name: http-zipkin
number: 15018
protocol: http
hosts:
- "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: zipkin-vs
spec:
hosts:
- "*"
gateways:
- zipkin-gateway
http:
- route:
- destination:
host: zipkin.istio-system.svc.cluster.local
port:
number: 9411
cluster1,cluster2: cm istio
[root@node01 ~]# kubectl get cm istio -n istio-system -o yaml
apiVersion: v1
data:
mesh: |-
accessLogFile: /dev/stdout
enableTracing: true
defaultConfig:
discoveryAddress: istiod.istio-system.svc:15012
meshId: mesh1
proxyMetadata: {}
tracing:
sampling: 100
zipkin:
address: 192.168.229.100:15018
enablePrometheusMerge: true
enableTracing: true
rootNamespace: istio-system
trustDomain: cluster.local
meshNetworks: 'networks: {}'
cluster1:
重启pod
kubectl rollout restart deploy -n istio
cluster2:
重启pod
kubectl rollout restart deploy -n istio
暴露服务:
kubectl port-forward --address 0.0.0.0 -n istio-system zipkin-6b8c6bdc56-m2b4f 9411:9411
清理:
cluster1:
kubectl delete secret istio-remote-secret-cluster2 -n istio-system
kubectl delete gw zipkin-gateway -n istio-system
kubectl delete vs zipkin-vs -n istio-system
istioctl x uninstall -f cluster1.yaml
reboot
cluster2:
kubectl delete secret istio-remote-secret-cluster1 -n istio-system
istioctl x uninstall -f cluster2.yaml
reboot
集群1
128,129,130
集群2
131,132,133
给istio-system namespace打标签 cluster1:
kubectl label namespace istio-system topology.istio.io/network=network1
cluster2:
kubectl label namespace istio-system topology.istio.io/network=network2
cluster1 生成istio operator部署文件
cat <<EOF > cluster1.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
profile: demo
values:
global:
meshID: mesh1
multiCluster:
clusterName: cluster1
network: network1
meshConfig:
accessLogFile: /dev/stdout
enableTracing: true
components:
egressGateways:
- name: istio-egressgateway
enabled: true
EOF
cluster2 生成istio operator部署文件
cat <<EOF > cluster2.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
profile: demo
values:
global:
meshID: mesh1
multiCluster:
clusterName: cluster2
network: network2
meshConfig:
accessLogFile: /dev/stdout
enableTracing: true
components:
egressGateways:
- name: istio-egressgateway
enabled: true
EOF
传输部署文件到cluster2
scp cluster2.yaml root@192.168.229.131:/root
生成监控apiserver secret
传输secret到cluster2
scp remote-secret-cluster1.yaml root@192.168.229.131:/root
cluster2 生成监控apiserver secret
istioctl x create-remote-secret --name=cluster2 --server=https://192.168.229.131:6443 > remote-secret-cluster2.yaml
传输secret到cluster1
scp remote-secret-cluster2.yaml root@192.168.229.128:/root
cluster1 部署监控apiserver secret
kubectl apply -f remote-secret-cluster2.yaml
部署istio
istioctl install -f cluster1.yaml
部署东西向网关
/root/istio-1.11.2/samples/multicluster/gen-eastwest-gateway.sh --mesh mesh1 --cluster cluster1 --network network1 | istioctl install -y -f -
配置东西向网关ip
kubectl patch svc -n istio-system istio-eastwestgateway -p '{"spec":{"externalIPs":["192.168.229.100"]}}'
暴露服务
kubectl apply -n istio-system -f /root/istio-1.11.2/samples/multicluster/expose-services.yaml
cluster2 部署监控apiserver secret
kubectl apply -f remote-secret-cluster1.yaml
部署istio
istioctl install -f cluster2.yaml
部署东西向网关
/root/istio-1.11.2/samples/multicluster/gen-eastwest-gateway.sh --mesh mesh1 --cluster cluster2 --network network2 | istioctl install -y -f -
配置东西向网关ip
kubectl patch svc -n istio-system istio-eastwestgateway -p '{"spec":{"externalIPs":["192.168.229.101"]}}'
暴露服务
kubectl apply -n istio-system -f /root/istio-1.11.2/samples/multicluster/expose-services.yaml
cluster1 重启pod
kubectl rollout restart deploy -n istio
kubectl rollout restart deploy -n istio-system
cluster2 重启pod
kubectl rollout restart deploy -n istio
kubectl rollout restart deploy -n istio-system
cluster1: 部署 zipkin
kubectl apply -f extras/zipkin.yaml -n istio-system
cluster1 : cm istio
[root@node01 ~]# kubectl get cm istio -n istio-system -o yaml
apiVersion: v1
data:
mesh: |-
accessLogFile: /dev/stdout
enableTracing: true
defaultConfig:
discoveryAddress: istiod.istio-system.svc:15012
meshId: mesh1
proxyMetadata: {}
tracing:
sampling: 100
zipkin:
address: zipkin.istio-system:9411
enablePrometheusMerge: true
enableTracing: true
rootNamespace: istio-system
trustDomain: cluster.local
meshNetworks: 'networks: {}'
cluster1: 暴露服务:
kubectl port-forward --address 0.0.0.0 -n istio-system zipkin-6b8c6bdc56-m2b4f 9411:9411
增加东西向网关端口 kubectl edit svc -n istio-system istio-eastwestgateway
- name: http-zipkin
nodePort: 32197
port: 15018
protocol: TCP
targetPort: 15018
暴露 zipkin 到 cluster2
visilazation/zipkin-gw-vs.yaml
kubectl apply -f zipkin-gw-vs.yaml -n istio-system
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: zipkin-gateway
spec:
selector:
istio: eastwestgateway
servers:
- port:
name: http-zipkin
number: 15018
protocol: http
hosts:
- "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: zipkin-vs
spec:
hosts:
- "*"
gateways:
- zipkin-gateway
http:
- route:
- destination:
host: zipkin.istio-system.svc.cluster.local
port:
number: 9411
cluster2 : cm istio
[root@node01 ~]# kubectl get cm istio -n istio-system -o yaml
apiVersion: v1
data:
mesh: |-
accessLogFile: /dev/stdout
enableTracing: true
defaultConfig:
discoveryAddress: istiod.istio-system.svc:15012
meshId: mesh1
proxyMetadata: {}
tracing:
sampling: 100
zipkin:
address: 192.168.229.100:15018
enablePrometheusMerge: true
enableTracing: true
rootNamespace: istio-system
trustDomain: cluster.local
meshNetworks: 'networks: {}'
清理:
cluster1:
kubectl label namespace istio-system topology.istio.io/network-
kubectl delete gw cross-network-gateway -n istio-system
kubectl delete secret istio-remote-secret-cluster2 -n istio-system
kubectl delete gw zipkin-gateway -n istio-system
kubectl delete vs zipkin-vs -n istio-system
istioctl x uninstall -f cluster1.yaml
reboot
cluster2:
kubectl label namespace istio-system topology.istio.io/network-
kubectl delete gw cross-network-gateway -n istio-system
kubectl delete secret istio-remote-secret-cluster1 -n istio-system
istioctl x uninstall -f cluster2.yaml
reboot
三个网络联通
集群1
137,138,139
集群2
140,141,142
集群3
143,144,145
网络联通
137,138,139
route add -net 172.21.2.0 netmask 255.255.255.0 gw 192.168.229.142
route add -net 172.21.0.0 netmask 255.255.255.0 gw 192.168.229.141
route add -net 172.21.1.0 netmask 255.255.255.0 gw 192.168.229.140
route add -net 172.22.2.0 netmask 255.255.255.0 gw 192.168.229.145
route add -net 172.22.0.0 netmask 255.255.255.0 gw 192.168.229.144
route add -net 172.22.1.0 netmask 255.255.255.0 gw 192.168.229.143
route add -net 10.70.0.0 netmask 255.255.0.0 gw 192.168.229.143
route add -net 10.69.0.0 netmask 255.255.0.0 gw 192.168.229.140
140,141,142
route add -net 172.20.2.0 netmask 255.255.255.0 gw 192.168.229.139
route add -net 172.20.0.0 netmask 255.255.255.0 gw 192.168.229.138
route add -net 172.20.1.0 netmask 255.255.255.0 gw 192.168.229.137
route add -net 172.22.2.0 netmask 255.255.255.0 gw 192.168.229.145
route add -net 172.22.0.0 netmask 255.255.255.0 gw 192.168.229.144
route add -net 172.22.1.0 netmask 255.255.255.0 gw 192.168.229.143
route add -net 10.70.0.0 netmask 255.255.0.0 gw 192.168.229.143
route add -net 10.68.0.0 netmask 255.255.0.0 gw 192.168.229.137
143,144,145
route add -net 172.21.2.0 netmask 255.255.255.0 gw 192.168.229.142
route add -net 172.21.0.0 netmask 255.255.255.0 gw 192.168.229.141
route add -net 172.21.1.0 netmask 255.255.255.0 gw 192.168.229.140
route add -net 172.20.2.0 netmask 255.255.255.0 gw 192.168.229.139
route add -net 172.20.0.0 netmask 255.255.255.0 gw 192.168.229.138
route add -net 172.20.1.0 netmask 255.255.255.0 gw 192.168.229.137
route add -net 10.69.0.0 netmask 255.255.0.0 gw 192.168.229.140
route add -net 10.68.0.0 netmask 255.255.0.0 gw 192.168.229.137
cluster1: 生成istio operator部署文件
cat <<EOF > cluster1.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
profile: demo
values:
global:
meshID: mesh1
multiCluster:
clusterName: cluster1
network: network1
meshConfig:
accessLogFile: /dev/stdout
enableTracing: true
components:
egressGateways:
- name: istio-egressgateway
enabled: true
EOF
这里我设置的cluster1东西向网关的ip试192.168.229.100 如果用的是loadblance,可以用下面命令获取
# export DISCOVERY_ADDRESS=$(kubectl -n istio-system get svc istio-eastwestgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
然后替换remotePilotAddress
生成istio operator部署文件
cat <<EOF > cluster2.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
profile: demo
values:
global:
meshID: mesh1
multiCluster:
clusterName: cluster2
network: network1
remotePilotAddress: 192.168.229.100
meshConfig:
accessLogFile: /dev/stdout
enableTracing: true
components:
egressGateways:
- name: istio-egressgateway
enabled: true
EOF
传输部署文件到cluster2
scp cluster2.yaml root@192.168.229.140:/root
这里我设置的cluster1东西向网关的ip试192.168.229.100 如果用的是loadblance,可以用下面命令获取
# export DISCOVERY_ADDRESS=$(kubectl -n istio-system get svc istio-eastwestgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
然后替换remotePilotAddress
生成istio operator部署文件
cat <<EOF > cluster3.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
profile: demo
values:
global:
meshID: mesh1
multiCluster:
clusterName: cluster3
network: network1
remotePilotAddress: 192.168.229.100
meshConfig:
accessLogFile: /dev/stdout
enableTracing: true
components:
egressGateways:
- name: istio-egressgateway
enabled: true
EOF
传输部署文件到cluster3
scp cluster3.yaml root@192.168.229.143:/root
部署istio
istioctl install -f cluster1.yaml
生成东西向网关
/root/istio-1.11.2/samples/multicluster/gen-eastwest-gateway.sh --mesh mesh1 --cluster cluster1 --network network1 | istioctl install -y -f -
配置东西向网关ip
kubectl patch svc -n istio-system istio-eastwestgateway -p '{"spec":{"externalIPs":["192.168.229.100"]}}'
暴露istiod
kubectl apply -n istio-system -f /root/istio-1.11.2/samples/multicluster/expose-istiod.yaml
cluster2: 生成访问apiserver secret
istioctl x create-remote-secret --name=cluster2 --server=https://192.168.229.140:6443 > remote-secret-cluster2.yaml
传输secret到cluster1
scp remote-secret-cluster2.yaml root@192.168.229.137:/root
cluster3: 生成访问apiserver secret
istioctl x create-remote-secret --name=cluster3 --server=https://192.168.229.143:6443 > remote-secret-cluster3.yaml
传输secret到cluster1
scp remote-secret-cluster3.yaml root@192.168.229.137:/root
cluster1 应用secret
kubectl apply -f remote-secret-cluster2.yaml
kubectl apply -f remote-secret-cluster3.yaml
cluster2: 部署istio
istioctl install -f cluster2.yaml
cluster3: 部署istio
istioctl install -f cluster3.yaml
cluster1: 重启pod
kubectl rollout restart deploy -n istio
kubectl rollout restart deploy -n istio-system
cluster2: 重启pod
kubectl rollout restart deploy -n istio
kubectl rollout restart deploy -n istio-system
cluster3: 重启pod
kubectl rollout restart deploy -n istio
kubectl rollout restart deploy -n istio-system
cluster1: 部署 zipkin
kubectl apply -f extras/zipkin.yaml -n istio-system
cluster1 增加东西向网关端口
kubectl patch svc -n istio-system istio-eastwestgateway -p '{"spec":{"ports":[{"name": "http-zipkin", "nodePort": 32197,"port": 15018, "protocol": "TCP", "targetPort": 15018}]}}'
cluster1:
暴露 zipkin
visilazation/zipkin-gw-vs.yaml
kubectl apply -f zipkin-gw-vs.yaml -n istio-system
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: zipkin-gateway
spec:
selector:
istio: eastwestgateway
servers:
- port:
name: http-zipkin
number: 15018
protocol: http
hosts:
- "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: zipkin-vs
spec:
hosts:
- "*"
gateways:
- zipkin-gateway
http:
- route:
- destination:
host: zipkin.istio-system.svc.cluster.local
port:
number: 9411
cluster1,cluster2,cluster3: cm istio
[root@node01 ~]# kubectl get cm istio -n istio-system -o yaml
apiVersion: v1
data:
mesh: |-
accessLogFile: /dev/stdout
enableTracing: true
defaultConfig:
discoveryAddress: istiod.istio-system.svc:15012
meshId: mesh1
proxyMetadata: {}
tracing:
sampling: 100
zipkin:
address: 192.168.229.100:15018
enablePrometheusMerge: true
enableTracing: true
rootNamespace: istio-system
trustDomain: cluster.local
meshNetworks: 'networks: {}'
修改
sampling: 100
zipkin:
address: 192.168.229.100:15018
cluster1:
重启pod
kubectl rollout restart deploy -n istio
cluster2:
重启pod
kubectl rollout restart deploy -n istio
cluster3:
重启pod
kubectl rollout restart deploy -n istio
我的集群的应用部署情况:
cluster1:
[root@node01 istio-teaching]# kubectl get pod -n istio
NAME READY STATUS RESTARTS AGE
productpage-v1-655c9d8c9-dln7x 2/2 Running 0 2m50s
ratings-v1-86ccf5754f-bz867 2/2 Running 0 2m50s
cluster2:
[root@node01 ~]# kubectl get pod -n istio
NAME READY STATUS RESTARTS AGE
reviews-v2-77f86758bd-9fb4n 2/2 Running 0 11m
cluster3:
[root@node01 ~]# kubectl get pod -n istio
NAME READY STATUS RESTARTS AGE
details-v1-548fbfb4d5-2xhkk 2/2 Running 0 11m
ratings-v1-678964777c-wkg4c 2/2 Running 0 11m
reviews-v3-76857cf4bf-5vhck 2/2 Running 0 11m
暴露服务:
kubectl port-forward --address 0.0.0.0 -n istio-system zipkin-6b8c6bdc56-m2b4f 9411:9411
cluster1:
kubectl delete secret istio-remote-secret-cluster2 -n istio-system
kubectl delete secret istio-remote-secret-cluster3 -n istio-system
kubectl delete gw zipkin-gateway -n istio-system
kubectl delete vs zipkin-vs -n istio-system
kubectl delete vs istiod-vs -n istio-system
kubectl delete gw istiod-gateway -n istio-system
istioctl x uninstall -f cluster1.yaml
reboot
cluster2:
istioctl x uninstall -f cluster2.yaml
reboot
cluster3:
istioctl x uninstall -f cluster3.yaml
reboot
两个网络 network2 东西向网管可以在cluster2也可以在cluster3 cluster2有网关,cluster3没有网关 不建议使用,按地域负载均衡的时候会有问题
集群1
137,138,139
集群2
140,141,142
集群3
143,144,145
打通cluster2,cluster3网络
140,141,142
route add -net 172.22.2.0 netmask 255.255.255.0 gw 192.168.229.145
route add -net 172.22.0.0 netmask 255.255.255.0 gw 192.168.229.144
route add -net 172.22.1.0 netmask 255.255.255.0 gw 192.168.229.143
route add -net 10.70.0.0 netmask 255.255.0.0 gw 192.168.229.143
143,144,145
route add -net 172.21.2.0 netmask 255.255.255.0 gw 192.168.229.142
route add -net 172.21.0.0 netmask 255.255.255.0 gw 192.168.229.141
route add -net 172.21.1.0 netmask 255.255.255.0 gw 192.168.229.140
route add -net 10.69.0.0 netmask 255.255.0.0 gw 192.168.229.140
给isito-system namespace打标签
cluster1:
kubectl label namespace istio-system topology.istio.io/network=network1
cluster1:
kubectl label namespace istio-system topology.istio.io/network=network2
cluster1:
kubectl label namespace istio-system topology.istio.io/network=network2
生成operator部署文件
cluster1:
cat <<EOF > cluster1.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
profile: demo
values:
global:
imagePullPolicy: IfNotPresent
meshID: mesh1
multiCluster:
clusterName: cluster1
network: network1
meshConfig:
accessLogFile: /dev/stdout
enableTracing: true
components:
egressGateways:
- name: istio-egressgateway
enabled: true
EOF
这里我设置的cluster1东西向网关的ip试192.168.229.100 如果用的是loadblance,可以用下面命令获取
export DISCOVERY_ADDRESS=$(kubectl -n istio-system get svc istio-eastwestgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
然后替换remotePilotAddress
生成operator部署文件
cat <<EOF > cluster2.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
profile: demo
values:
global:
imagePullPolicy: IfNotPresent
meshID: mesh1
multiCluster:
clusterName: cluster2
network: network2
remotePilotAddress: 192.168.229.100
meshConfig:
accessLogFile: /dev/stdout
enableTracing: true
components:
egressGateways:
- name: istio-egressgateway
enabled: true
EOF
这里我设置的cluster1东西向网关的ip试192.168.229.100 如果用的是loadblance,可以用下面命令获取
export DISCOVERY_ADDRESS=$(kubectl -n istio-system get svc istio-eastwestgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
然后替换remotePilotAddress
生成operator部署文件
cat <<EOF > cluster3.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
profile: demo
values:
global:
imagePullPolicy: IfNotPresent
meshID: mesh1
multiCluster:
clusterName: cluster3
network: network2
remotePilotAddress: 192.168.229.100
meshConfig:
accessLogFile: /dev/stdout
enableTracing: true
components:
egressGateways:
- name: istio-egressgateway
enabled: true
EOF
把部署文件传到cluster2
scp cluster2.yaml root@192.168.229.140:/root
把部署文件传到cluster3
scp cluster3.yaml root@192.168.229.143:/root
部署cluster1
istioctl install -f cluster1.yaml
部署东西向网关
/root/istio-1.11.2/samples/multicluster/gen-eastwest-gateway.sh --mesh mesh1 --cluster cluster1 --network network1 | istioctl install -y -f -
配置东西向网关ip
kubectl patch svc -n istio-system istio-eastwestgateway -p '{"spec":{"externalIPs":["192.168.229.100"]}}'
暴露istiod
kubectl apply -n istio-system -f /root/istio-1.11.2/samples/multicluster/expose-istiod.yaml
暴露服务
kubectl apply -n istio-system -f /root/istio-1.11.2/samples/multicluster/expose-services.yaml
cluster2: 生成监控apiserver secret
istioctl x create-remote-secret --name=cluster2 --server=https://192.168.229.140:6443 > remote-secret-cluster2.yaml
传输secret到cluster1
scp remote-secret-cluster2.yaml root@192.168.229.137:/root
cluster3: 生成监控apiserver secret
istioctl x create-remote-secret --name=cluster3 --server=https://192.168.229.143:6443 > remote-secret-cluster3.yaml
传输secret到cluster1
scp remote-secret-cluster3.yaml root@192.168.229.137:/root
cluster1: 应用监控apiserver secret
kubectl apply -f remote-secret-cluster2.yaml
kubectl apply -f remote-secret-cluster3.yaml
cluster2: 部署cluster2
istioctl install -f cluster2.yaml
安装东西向网关
/root/istio-1.11.2/samples/multicluster/gen-eastwest-gateway.sh --mesh mesh1 --cluster cluster2 --network network2 | istioctl install -y -f -
配置东西向网关ip
kubectl patch svc -n istio-system istio-eastwestgateway -p '{"spec":{"externalIPs":["192.168.229.101"]}}'
暴露服务
kubectl apply -n istio-system -f /root/istio-1.11.2/samples/multicluster/expose-services.yaml
cluster3: 部署cluster3
istioctl install -f cluster3.yaml
cluster1: 重启pod kubectl rollout restart deploy -n istio kubectl rollout restart deploy -n istio-system
cluster2: 重启pod kubectl rollout restart deploy -n istio kubectl rollout restart deploy -n istio-system
cluster3: 重启pod kubectl rollout restart deploy -n istio kubectl rollout restart deploy -n istio-system
cluster1: 部署 zipkin
kubectl apply -f extras/zipkin.yaml -n istio-system
cluster1 增加东西向网关端口
kubectl patch svc -n istio-system istio-eastwestgateway -p '{"spec":{"ports":[{"name": "http-zipkin", "nodePort": 32197,"port": 15018, "protocol": "TCP", "targetPort": 15018}]}}'
cluster1:
暴露 zipkin
visilazation/zipkin-gw-vs.yaml
kubectl apply -f zipkin-gw-vs.yaml -n istio-system
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: zipkin-gateway
spec:
selector:
istio: eastwestgateway
servers:
- port:
name: http-zipkin
number: 15018
protocol: http
hosts:
- "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: zipkin-vs
spec:
hosts:
- "*"
gateways:
- zipkin-gateway
http:
- route:
- destination:
host: zipkin.istio-system.svc.cluster.local
port:
number: 9411
cluster1,cluster2,cluster3: cm istio
[root@node01 ~]# kubectl get cm istio -n istio-system -o yaml
apiVersion: v1
data:
mesh: |-
accessLogFile: /dev/stdout
enableTracing: true
defaultConfig:
discoveryAddress: istiod.istio-system.svc:15012
meshId: mesh1
proxyMetadata: {}
tracing:
sampling: 100
zipkin:
address: 192.168.229.100:15018
enablePrometheusMerge: true
enableTracing: true
rootNamespace: istio-system
trustDomain: cluster.local
meshNetworks: 'networks: {}'
修改
sampling: 100
zipkin:
address: 192.168.229.100:15018
cluster1:
重启pod
kubectl rollout restart deploy -n istio
cluster2:
重启pod
kubectl rollout restart deploy -n istio
cluster3:
重启pod
kubectl rollout restart deploy -n istio
暴露服务:
kubectl port-forward --address 0.0.0.0 -n istio-system zipkin-6b8c6bdc56-m2b4f 9411:9411
清理:
cluster1:
kubectl label namespace istio-system topology.istio.io/network-
kubectl delete secret istio-remote-secret-cluster2 -n istio-system
kubectl delete secret istio-remote-secret-cluster3 -n istio-system
kubectl delete gw zipkin-gateway -n istio-system
kubectl delete vs zipkin-vs -n istio-system
kubectl delete gw cross-network-gateway -n istio-system
kubectl delete gw istiod-gateway -n istio-system
kubectl delete vs istiod-vs -n istio-system
istioctl x uninstall -f cluster1.yaml
reboot
cluster2:
kubectl label namespace istio-system topology.istio.io/network-
kubectl delete gw cross-network-gateway -n istio-system
istioctl x uninstall -f cluster2.yaml
reboot
cluster3:
kubectl label namespace istio-system topology.istio.io/network-
istioctl x uninstall -f cluster3.yaml
reboot
两个网络
三个东西向网关
集群1
137,138,139
集群2
140,141,142
集群3
143,144,145
打通cluster2,cluster3网络
140,141,142
route add -net 172.22.2.0 netmask 255.255.255.0 gw 192.168.229.145
route add -net 172.22.0.0 netmask 255.255.255.0 gw 192.168.229.144
route add -net 172.22.1.0 netmask 255.255.255.0 gw 192.168.229.143
route add -net 10.70.0.0 netmask 255.255.0.0 gw 192.168.229.143
143,144,145
route add -net 172.21.2.0 netmask 255.255.255.0 gw 192.168.229.142
route add -net 172.21.0.0 netmask 255.255.255.0 gw 192.168.229.141
route add -net 172.21.1.0 netmask 255.255.255.0 gw 192.168.229.140
route add -net 10.69.0.0 netmask 255.255.0.0 gw 192.168.229.140
给isito-system namespace打标签
cluster1:
kubectl label namespace istio-system topology.istio.io/network=network1
cluster2:
kubectl label namespace istio-system topology.istio.io/network=network2
cluster3:
kubectl label namespace istio-system topology.istio.io/network=network2
生成operator部署文件 cluster1:
cat <<EOF > cluster1.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
profile: demo
values:
global:
imagePullPolicy: IfNotPresent
meshID: mesh1
multiCluster:
clusterName: cluster1
network: network1
meshConfig:
accessLogFile: /dev/stdout
enableTracing: true
components:
egressGateways:
- name: istio-egressgateway
enabled: true
EOF
这里我设置的cluster1东西向网关的ip试192.168.229.100 如果用的是loadblance,可以用下面命令获取
export DISCOVERY_ADDRESS=$(kubectl -n istio-system get svc istio-eastwestgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
然后替换remotePilotAddress
生成operator部署文件
cat <<EOF > cluster2.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
profile: demo
values:
global:
imagePullPolicy: IfNotPresent
meshID: mesh1
multiCluster:
clusterName: cluster2
network: network2
remotePilotAddress: 192.168.229.100
meshConfig:
accessLogFile: /dev/stdout
enableTracing: true
components:
egressGateways:
- name: istio-egressgateway
enabled: true
EOF
这里我设置的cluster1东西向网关的ip试192.168.229.100 如果用的是loadblance,可以用下面命令获取
export DISCOVERY_ADDRESS=$(kubectl -n istio-system get svc istio-eastwestgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
然后替换remotePilotAddress
生成operator部署文件
cat <<EOF > cluster3.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
profile: demo
values:
global:
imagePullPolicy: IfNotPresent
meshID: mesh1
multiCluster:
clusterName: cluster3
network: network2
remotePilotAddress: 192.168.229.100
meshConfig:
accessLogFile: /dev/stdout
enableTracing: true
components:
egressGateways:
- name: istio-egressgateway
enabled: true
EOF
把部署文件传到cluster2
scp cluster2.yaml root@192.168.229.140:/root
把部署文件传到cluster3
scp cluster3.yaml root@192.168.229.143:/root
部署cluster1
istioctl install -f cluster1.yaml
部署东西向网关
/root/istio-1.11.2/samples/multicluster/gen-eastwest-gateway.sh --mesh mesh1 --cluster cluster1 --network network1 | istioctl install -y -f -
配置东西向网关ip
kubectl patch svc -n istio-system istio-eastwestgateway -p '{"spec":{"externalIPs":["192.168.229.100"]}}'
暴露istiod
kubectl apply -n istio-system -f /root/istio-1.11.2/samples/multicluster/expose-istiod.yaml
暴露服务
kubectl apply -n istio-system -f /root/istio-1.11.2/samples/multicluster/expose-services.yaml
cluster2: 生成监控apiserver secret
istioctl x create-remote-secret --name=cluster2 --server=https://192.168.229.140:6443 > remote-secret-cluster2.yaml
传输secret到cluster1
scp remote-secret-cluster2.yaml root@192.168.229.137:/root
cluster3: 生成监控apiserver secret
istioctl x create-remote-secret --name=cluster3 --server=https://192.168.229.143:6443 > remote-secret-cluster3.yaml
传输secret到cluster1
scp remote-secret-clu
cluster1: 应用监控apiserver secret
kubectl apply -f remote-secret-cluster2.yaml
kubectl apply -f remote-secret-cluster3.yaml
cluster2: 部署cluster2
istioctl install -f cluster2.yaml
安装东西向网关
/root/istio-1.11.2/samples/multicluster/gen-eastwest-gateway.sh --mesh mesh1 --cluster cluster2 --network network2 | istioctl install -y -f -
配置东西向网关ip
kubectl patch svc -n istio-system istio-eastwestgateway -p '{"spec":{"externalIPs":["192.168.229.101"]}}'
暴露服务
kubectl apply -n istio-system -f /root/istio-1.11.2/samples/multicluster/expose-services.yaml
cluster3: 部署cluster3
istioctl install -f cluster3.yaml
安装东西向网关
/root/istio-1.11.2/samples/multicluster/gen-eastwest-gateway.sh --mesh mesh1 --cluster cluster3 --network network2 | istioctl install -y -f -
配置东西向网关ip
kubectl patch svc -n istio-system istio-eastwestgateway -p '{"spec":{"externalIPs":["192.168.229.102"]}}'
暴露服务
kubectl apply -n istio-system -f /root/istio-1.11.2/samples/multicluster/expose-services.yaml
cluster1:
重启pod
kubectl rollout restart deploy -n istio
kubectl rollout restart deploy -n istio-system
cluster2:
重启pod
kubectl rollout restart deploy -n istio
kubectl rollout restart deploy -n istio-system
cluster1:
重启pod
kubectl rollout restart deploy -n istio
kubectl rollout restart deploy -n istio-system
cluster1: 部署 zipkin
kubectl apply -f extras/zipkin.yaml -n istio-system
cluster1 增加东西向网关端口
kubectl patch svc -n istio-system istio-eastwestgateway -p '{"spec":{"ports":[{"name": "http-zipkin", "nodePort": 32197,"port": 15018, "protocol": "TCP", "targetPort": 15018}]}}'
cluster1:
暴露 zipkin
visilazation/zipkin-gw-vs.yaml
kubectl apply -f zipkin-gw-vs.yaml -n istio-system
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: zipkin-gateway
spec:
selector:
istio: eastwestgateway
servers:
- port:
name: http-zipkin
number: 15018
protocol: http
hosts:
- "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: zipkin-vs
spec:
hosts:
- "*"
gateways:
- zipkin-gateway
http:
- route:
- destination:
host: zipkin.istio-system.svc.cluster.local
port:
number: 9411
cluster1,cluster2,cluster3: cm istio
[root@node01 ~]# kubectl get cm istio -n istio-system -o yaml
apiVersion: v1
data:
mesh: |-
accessLogFile: /dev/stdout
enableTracing: true
defaultConfig:
discoveryAddress: istiod.istio-system.svc:15012
meshId: mesh1
proxyMetadata: {}
tracing:
sampling: 100
zipkin:
address: 192.168.229.100:15018
enablePrometheusMerge: true
enableTracing: true
rootNamespace: istio-system
trustDomain: cluster.local
meshNetworks: 'networks: {}'
修改
sampling: 100
zipkin:
address: 192.168.229.100:15018
cluster1:
重启pod
kubectl rollout restart deploy -n istio
cluster2:
重启pod
kubectl rollout restart deploy -n istio
cluster3:
重启pod
kubectl rollout restart deploy -n istio
暴露服务:
kubectl port-forward --address 0.0.0.0 -n istio-system zipkin-6b8c6bdc56-m2b4f 9411:9411
清理:
cluster1:
kubectl label namespace istio-system topology.istio.io/network-
kubectl delete gw zipkin-gateway -n istio-system
kubectl delete vs zipkin-vs -n istio-system
kubectl delete secret istio-remote-secret-cluster2 -n istio-system
kubectl delete secret istio-remote-secret-cluster3 -n istio-system
kubectl delete gw cross-network-gateway -n istio-system
kubectl delete gw istiod-gateway -n istio-system
kubectl delete vs istiod-vs -n istio-system
istioctl x uninstall -f cluster1.yaml
reboot
cluster2:
kubectl label namespace istio-system topology.istio.io/network-
kubectl delete gw cross-network-gateway -n istio-system
istioctl x uninstall -f cluster2.yaml
reboot
cluster3:
kubectl label namespace istio-system topology.istio.io/network-
kubectl delete gw cross-network-gateway -n istio-system
istioctl x uninstall -f cluster3.yaml
reboot
三个网络
集群1
137,138,139
集群2
140,141,142
集群3
143,144,145
给istio-system namespace打标签
cluster1:
kubectl label namespace istio-system topology.istio.io/network=network1
cluster2:
kubectl label namespace istio-system topology.istio.io/network=network2
cluster3:
kubectl label namespace istio-system topology.istio.io/network=network3
cluster1: 生成istio operator部署文件
cat <<EOF > cluster1.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
profile: demo
values:
global:
meshID: mesh1
multiCluster:
clusterName: cluster1
network: network1
meshConfig:
accessLogFile: /dev/stdout
enableTracing: true
components:
egressGateways:
- name: istio-egressgateway
enabled: true
EOF
这里我设置的cluster1东西向网关的ip试192.168.229.100 如果用的是loadblance,可以用下面命令获取
export DISCOVERY_ADDRESS=$(kubectl -n istio-system get svc istio-eastwestgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
然后替换remotePilotAddress
生成istio operator部署文件
cat <<EOF > cluster2.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
profile: demo
values:
global:
meshID: mesh1
multiCluster:
clusterName: cluster2
network: network2
remotePilotAddress: 192.168.229.100
meshConfig:
accessLogFile: /dev/stdout
enableTracing: true
components:
egressGateways:
- name: istio-egressgateway
enabled: true
EOF
这里我设置的cluster1东西向网关的ip试192.168.229.100 如果用的是loadblance,可以用下面命令获取
export DISCOVERY_ADDRESS=$(kubectl -n istio-system get svc istio-eastwestgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
然后替换remotePilotAddress
生成istio operator部署文件
cat <<EOF > cluster3.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
profile: demo
values:
global:
meshID: mesh1
multiCluster:
clusterName: cluster3
network: network3
remotePilotAddress: 192.168.229.100
meshConfig:
accessLogFile: /dev/stdout
enableTracing: true
components:
egressGateways:
- name: istio-egressgateway
enabled: true
EOF
传输部署文件到cluster2
scp cluster2.yaml root@192.168.229.140:/root
传输部署文件到cluster3
scp cluster3.yaml root@192.168.229.143:/root
安装istio
istioctl install -f cluster1.yaml
生成东西向网关
/root/istio-1.11.2/samples/multicluster/gen-eastwest-gateway.sh --mesh mesh1 --cluster cluster1 --network network1 | istioctl install -y -f -
配置东西向网关ip
kubectl patch svc -n istio-system istio-eastwestgateway -p '{"spec":{"externalIPs":["192.168.229.100"]}}'
暴露istiod
kubectl apply -n istio-system -f /root/istio-1.11.2/samples/multicluster/expose-istiod.yaml
暴露service
kubectl apply -n istio-system -f /root/istio-1.11.2/samples/multicluster/expose-services.yaml
cluster2: 生成访问apiserver的secret
istioctl x create-remote-secret --name=cluster2 --server=https://192.168.229.140:6443 > remote-secret-cluster2.yaml
传输secret到cluster1
scp remote-secret-cluster2.yaml root@192.168.229.137:/root
cluster3: 生成访问apiserver的secret
istioctl x create-remote-secret --name=cluster3 --server=https://192.168.229.143:6443 > remote-secret-cluster3.yaml
传输secret到cluster1
scp remote-secret-cluster3.yaml root@192.168.229.137:/root
cluster1: 应用secret
kubectl apply -f remote-secret-cluster2.yaml
kubectl apply -f remote-secret-cluster3.yaml
cluster2: 部署istio
istioctl install -f cluster2.yaml
生成东西向网关
/root/istio-1.11.2/samples/multicluster/gen-eastwest-gateway.sh --mesh mesh1 --cluster cluster2 --network network2 | istioctl install -y -f -
配置东西向网关ip
kubectl patch svc -n istio-system istio-eastwestgateway -p '{"spec":{"externalIPs":["192.168.229.101"]}}'
暴露service
kubectl apply -n istio-system -f /root/istio-1.11.2/samples/multicluster/expose-services.yaml
cluster3: 部署istio
istioctl install -f cluster3.yaml
生成东西向网关
/root/istio-1.11.2/samples/multicluster/gen-eastwest-gateway.sh --mesh mesh1 --cluster cluster3 --network network3 | istioctl install -y -f -
配置东西向网关ip
kubectl patch svc -n istio-system istio-eastwestgateway -p '{"spec":{"externalIPs":["192.168.229.102"]}}'
暴露service
kubectl apply -n istio-system -f /root/istio-1.11.2/samples/multicluster/expose-services.yaml
cluster1:
重启pod
kubectl rollout restart deploy -n istio
kubectl rollout restart deploy -n istio-system
cluster2:
重启pod
kubectl rollout restart deploy -n istio
kubectl rollout restart deploy -n istio-system
cluster3:
重启pod
kubectl rollout restart deploy -n istio
kubectl rollout restart deploy -n istio-system
cluster1: 部署 zipkin
kubectl apply -f extras/zipkin.yaml -n istio-system
cluster1 增加东西向网关端口
kubectl patch svc -n istio-system istio-eastwestgateway -p '{"spec":{"ports":[{"name": "http-zipkin", "nodePort": 32197,"port": 15018, "protocol": "TCP", "targetPort": 15018}]}}'
cluster1:
暴露 zipkin
visilazation/zipkin-gw-vs.yaml
kubectl apply -f zipkin-gw-vs.yaml -n istio-system
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: zipkin-gateway
spec:
selector:
istio: eastwestgateway
servers:
- port:
name: http-zipkin
number: 15018
protocol: http
hosts:
- "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: zipkin-vs
spec:
hosts:
- "*"
gateways:
- zipkin-gateway
http:
- route:
- destination:
host: zipkin.istio-system.svc.cluster.local
port:
number: 9411
cluster1,cluster2,cluster3: cm istio
[root@node01 ~]# kubectl get cm istio -n istio-system -o yaml
apiVersion: v1
data:
mesh: |-
accessLogFile: /dev/stdout
enableTracing: true
defaultConfig:
discoveryAddress: istiod.istio-system.svc:15012
meshId: mesh1
proxyMetadata: {}
tracing:
sampling: 100
zipkin:
address: 192.168.229.100:15018
enablePrometheusMerge: true
enableTracing: true
rootNamespace: istio-system
trustDomain: cluster.local
meshNetworks: 'networks: {}'
修改
sampling: 100
zipkin:
address: 192.168.229.100:15018
cluster1:
重启pod
kubectl rollout restart deploy -n istio
cluster2:
重启pod
kubectl rollout restart deploy -n istio
cluster3:
重启pod
kubectl rollout restart deploy -n istio
暴露服务:
kubectl port-forward --address 0.0.0.0 -n istio-system zipkin-6b8c6bdc56-m2b4f 9411:9411
清理:
cluster1:
kubectl label namespace istio-system topology.istio.io/network-
kubectl delete secret istio-remote-secret-cluster2 -n istio-system
kubectl delete secret istio-remote-secret-cluster3 -n istio-system
kubectl delete gw cross-network-gateway -n istio-system
kubectl delete gw istiod-gateway -n istio-system
kubectl delete vs istiod-vs -n istio-system
kubectl delete gw zipkin-gateway -n istio-system
kubectl delete vs zipkin-vs -n istio-system
istioctl x uninstall -f cluster1.yaml
reboot
cluster2:
kubectl label namespace istio-system topology.istio.io/network-
kubectl delete gw cross-network-gateway -n istio-system
istioctl x uninstall -f cluster2.yaml
reboot
cluster3:
kubectl label namespace istio-system topology.istio.io/network-
kubectl delete gw cross-network-gateway -n istio-system
istioctl x uninstall -f cluster3.yaml
reboot
- EOF -
2、云原生思想
Go 开发大全
参与维护一个非常全面的Go开源技术资源库。日常分享 Go, 云原生、k8s、Docker和微服务方面的技术文章和行业动态。
关注后获取
回复 Go 获取6万star的Go资源库
分享、点赞和在看
支持我们分享更多好文章,谢谢!