istio流量劫持总结
§ envoy 启动流程
数据面组件启动流程
initContainer执行初始化脚本,为Pod添加iptables规则
Pilot-agent根据启动参数和K8S API Server中的配置信息生成Envoy的初始配置文件envoy-rev0.json,该文件告诉Envoy从xDS server中获取动态配置信息,并配置了xDS server的地址信息,即控制面的Pilot。
Pilot-agent使用envoy-rev0.json启动Envoy进程。
Envoy根据初始配置获得Pilot地址,采用xDS接口从Pilot获取到Listener,Cluster,Route等d动态配置信息。
Envoy根据获取到的动态配置启动Listener,并根据Listener的配置,结合Route和Cluster对拦截到的流量进行处理。
§ istio劫持流量总流程
进流量:
downstream -> iptables -> envoy Inbound -> app
出流量:
app -> iptables -> envoy Outbound -> upstream
§ 1. IPTABLES 劫持转发流量的过程
启动流程中有讲到第一步是envoy执行初始化脚本,添加iptables规则
参考文章
[istio sidecar中创建iptables流程
](https://www.jianshu.com/p/50af80c5baba)
§ 2. envoy Inbound 和 Outbound 流程
§ istio 的监听端口
$ nsenter -t 20533 -n netstat -nl
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 0.0.0.0:15090 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:15000 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:15001 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:15006 0.0.0.0:* LISTEN
tcp6 0 0 :::15020 :::* LISTEN
tcp6 0 0 :::8889 :::* LISTEN
由上图可以看到envoy有多个端口在监听
15000 端口: 管理端口
15001 端口:Outbound流量重定向端口
15006 端口:Inbound流量重定向端口
15020 端口:健康检查端口
15090 端口: 普罗米修斯端口
应用端口:8899
§ envoy Inbound处理流量过程
Inbound 流程是将 iptables 拦截到的 downstream 的流量转交给 localhost,与 Pod 内的应用程序容器建立连接。
下面以进入calculator的流量为例子,来分析一下Inbound流程
§ calculator Inbound流程图
§ Inbound配置
listener
virtual listener
{
"version_info": "2020-02-11T01:52:11Z/5",
"listener": {
"name": "virtualInbound",
"address": {
"socket_address": {
"address": "0.0.0.0",
"port_value": 15006
}
},
"filter_chains": [{
"filter_chain_match": {
"prefix_ranges": [{
"address_prefix": "::0",
"prefix_len": 0
}]
},
"filters": [{
"name": "envoy.tcp_proxy",
"typed_config": {
"@type": "type.googleapis.com/envoy.config.filter.network.tcp_proxy.v2.TcpProxy",
"stat_prefix": "InboundPassthroughClusterIpv6",
"cluster": "InboundPassthroughClusterIpv6"
}
}],
"filter_chain_match": {
"prefix_ranges": [{
"address_prefix": "192.144.99.163",
"prefix_len": 32
}],
"destination_port": 8888
},
"filters": [{
"name": "envoy.http_connection_manager",
"typed_config": {
"@type": "type.googleapis.com/envoy.config.filter.network.http_connection_manager.v2.HttpConnectionManager",
"stat_prefix": "inbound_192.144.99.163_8888",
"http_filters": [{
"name": "istio_authn"
},
{
"name": "mixer"
},
{
"name": "envoy.router"
}
],
"route_config": {
"name": "inbound|8888|grpc|calculator.vm.svc.cluster.local",
"virtual_hosts": [{
"name": "inbound|http|8888",
"domains": [
"*"
],
"routes": [{
"match": {
"prefix": "/"
},
"decorator": {
"operation": "calculator.vm.svc.cluster.local:8888/*"
},
"name": "default",
"route": {
"timeout": "0s",
"max_grpc_timeout": "0s",
"cluster": "inbound|8888|grpc|calculator.vm.svc.cluster.local"
}
}]
}],
"validate_clusters": false
}
}
}]
}]
}
}
该Listener中第三个filterchain用于处理Calculator服务的入向请求。
该filterchain的匹配条件为Calculator服务的 IP和8888端口,配置了一个http_connection_manager filter
http_connection_manager 中又嵌入了istio_auth,Mixer,envoy.router等http filter,经过这些filter进行处理后,请求最终将被转发给"“cluster”: “inbound|8888|grpc|calculator.vm.svc.cluster.local”"
接下来我们看下inbound cluster的配置
cluster
inbound cluster
{
"version_info": "2020-02-10T08:16:25Z/3",
"cluster": {
"name": "inbound|8888|grpc|calculator.vm.svc.cluster.local",
"type": "STATIC",
"connect_timeout": "10s",
"circuit_breakers": {
"thresholds": [
{
"max_connections": 4294967295,
"max_pending_requests": 4294967295,
"max_requests": 4294967295,
"max_retries": 4294967295
}
]
},
"http2_protocol_options": {
"max_concurrent_streams": 1073741824
},
"load_assignment": {
"cluster_name": "inbound|8888|grpc|calculator.vm.svc.cluster.local",
"endpoints": [
{
"lb_endpoints": [
{
"endpoint": {
"address": {
"socket_address": {
"address": "127.0.0.1",
"port_value": 8888
}
}
}
}
]
}
]
}
}
}
这个Inbound Cluster,由于该Inbound Cluster中配置的Upstream为127.0.0.1:8888(?如何查看其upstream)
由于iptable设置中127.0.0.1不会被拦截,该请求将发送到Calculator服务的8888端口上进行业务处理。
§ Inbound流程代码
//todo
§ envoy Outbound处理流量过程
§ webapp outbound流程图
§ outbound配置
virtual outbound istener
{
"listener": {
"name": "virtualOutbound",
"address": {
"socket_address": {
"address": "0.0.0.0",
"port_value": 15001
}
},
"filter_chains": [{
"filter_chain_match": {
"prefix_ranges": [{
"address_prefix": "10.244.2.73",
"prefix_len": 32
}]
},
"filters": [{
"name": "envoy.tcp_proxy",
"typed_config": {
"@type": "type.googleapis.com/envoy.config.filter.network.tcp_proxy.v2.TcpProxy",
"stat_prefix": "BlackHoleCluster",
"cluster": "BlackHoleCluster"
}
}]
},
{
"filters": [{
"name": "mixer"
},
{
"name": "envoy.tcp_proxy",
"typed_config": {
"@type": "type.googleapis.com/envoy.config.filter.network.tcp_proxy.v2.TcpProxy",
"stat_prefix": "PassthroughCluster",
"cluster": "PassthroughCluster"
}
}
]
}
],
"use_original_dst": true
}
}
改listener的最后一行有个配置 “use_original_dst”: true,表示用原有地址,不进行路由,这样就会转发到0.0.0.0_8888这个监听器上
下面看一下0.0.0.0_8888这个监听器的配置
outbound listener
{
"listener": {
"name": "0.0.0.0_8888",
"address": {
"socket_address": {
"address": "0.0.0.0",
"port_value": 8888
}
},
"filter_chains": [{
"filter_chain_match": {
"prefix_ranges": [{
"address_prefix": "10.244.2.73",
"prefix_len": 32
}]
},
"filters": [{
"name": "envoy.tcp_proxy",
"typed_config": {
"@type": "type.googleapis.com/envoy.config.filter.network.tcp_proxy.v2.TcpProxy",
"stat_prefix": "BlackHoleCluster",
"cluster": "BlackHoleCluster"
}
}]
},
{
"filters": [{
"name": "envoy.http_connection_manager",
"typed_config": {
"@type": "type.googleapis.com/envoy.config.filter.network.http_connection_manager.v2.HttpConnectionManager",
"stat_prefix": "outbound_0.0.0.0_8888",
"http_filters": [{
"name": "mixer"
},
{
"name": "envoy.cors"
},
{
"name": "envoy.fault"
},
{
"name": "envoy.router"
}
],
"use_remote_address": false,
"generate_request_id": true,
"upgrade_configs": [{
"upgrade_type": "websocket"
}],
"stream_idle_timeout": "0s",
"normalize_path": true,
"rds": {
"config_source": {
"ads": {}
},
"route_config_name": "8888"
}
}
}]
}
],
"deprecated_v1": {
"bind_to_port": false
},
"listener_filters_timeout": "0.100s",
"traffic_direction": "OUTBOUND",
"continue_on_listener_filters_timeout": true
}
}
同理最后有一个envoy.router filter,这个filter中route_config_name:“8888”,所以我们可以找到名为"8888"的Route配置
outbound Route
{
"route_config": {
"name": "8888",
"virtual_hosts": [{
"name": "calculator.vm.svc.cluster.local:8888",
"domains": [
"calculator.vm.svc.cluster.local",
"calculator.vm.svc.cluster.local:8888",
"calculator.vm",
"calculator.vm:8888",
"calculator.vm.svc.cluster",
"calculator.vm.svc.cluster:8888",
"calculator.vm.svc",
"calculator.vm.svc:8888",
"10.106.45.142",
"10.106.45.142:8888"
],
"routes": [{
"match": {
"prefix": "/"
},
"route": {
"weighted_clusters": {
"clusters": [{
"name": "outbound|8888|v1|calculator.vm.svc.cluster.local",
"weight": 50
},
{
"name": "outbound|8888|v2|calculator.vm.svc.cluster.local",
"weight": 50
}
]
}
},
"decorator": {
"operation": "calculator:8888/*"
}
}]
},
{
"name": "allow_any",
"domains": [
"*"
],
"routes": [{
"match": {
"prefix": "/"
},
"route": {
"cluster": "PassthroughCluster"
}
}]
}
],
"validate_clusters": false
}
}
根据Route 8888的配置可以找到outbound cluster
outbound cluster
{
"version_info": "2020-02-07T10:38:17Z/1847",
"cluster": {
"name": "outbound|8888|v2|calculator.vm.svc.cluster.local",
"type": "EDS",
"eds_cluster_config": {
"eds_config": {
"ads": {}
},
"service_name": "outbound|8888|v2|calculator.vm.svc.cluster.local"
},
"connect_timeout": "10s",
"circuit_breakers": {
"thresholds": [
{
"max_connections": 4294967295,
"max_pending_requests": 4294967295,
"max_requests": 4294967295,
"max_retries": 4294967295
}
]
},
"http2_protocol_options": {
"max_concurrent_streams": 1073741824
},
"metadata": {
"filter_metadata": {
"istio": {
"config": "/apis/networking/v1alpha3/namespaces/vm/destination-rule/calculator"
}
}
}
},
"last_updated": "2020-02-07T10:38:17.893Z"
}
我们会发现这个cluster没有直接的endpoint,因为这是动态资源,
可以通过Pilot的调试接口获取该Cluster的endpoint:
$ curl http://xxx.xxx.xxx.xxx.108:15014/debug/edsz > pilot_eds_dump
最后发现endpoint为192.xxx.xxx.163:8888
然后就把流量转发到calculator服务了
BlackHoleCluster
这是一个特殊的Cluster,并没有配置后端处理请求的Host。如其名字所暗示的一样,请求进入后将被直接丢弃掉。如果一个请求没有找到其对的目的服务,则被发到cluste。
{
"cluster": {
"name": "BlackHoleCluster",
"type": "STATIC",
"connect_timeout": "1s"
}
}
*PassthroughCluster
和BlackHoleCluter相反,发向PassthroughCluster的请求会被直接发送到其请求中要求的原始目地的,Envoy不会对请求进行重新路由。
{
"cluster": {
"name": "PassthroughCluster",
"type": "ORIGINAL_DST",
"connect_timeout": "1s",
"lb_policy": "CLUSTER_PROVIDED",
"circuit_breakers": {
"thresholds": [
{
"max_connections": 4294967295,
"max_pending_requests": 4294967295,
"max_requests": 4294967295,
"max_retries": 4294967295
}
]
}
}
}
§ outbound流程代码
//todo
- 点赞
- 收藏
- 关注作者
评论(0)