istio sidecar中创建iptables流程
§ istio-iptables.sh 源码解析
§ 启动参数解析代码
istio的init container中初始化iptables的命令如下:
istio-iptables -p 15001 -z 15006 -u 1337 -m REDIRECT -i * -x -b * -d 15020
这条命令的意思是什么呢?
以下是istio-iptables.sh中的usage函数,表明了每个参数一意思以及默认值,看完这个就可以很清楚的明白这条命令的意思
其实就是让Envoy代理拦截所有进出Pod的流量,将入站流量重定向到Envoy的15006端口(Inbound端口),将出站流量重定向到15001端口(Outbound端口),同时排除15020端口不进行重定向(健康检查端口)
function usage() {
echo "${0} -p PORT -u UID -g GID [-m mode] [-b ports] [-d ports] [-i CIDR] [-x CIDR] [-k interfaces] [-t] [-h]"
echo ''
# shellcheck disable=SC2016
echo ' -p: Specify the envoy port to which redirect all TCP traffic (default $ENVOY_PORT = 15001)'
# shellcheck disable=SC2016
echo ' -z: Port to which all inbound TCP traffic to the pod/VM should be redirected to. For REDIRECT only (default $INBOUND_CAPTURE_PORT = 15006)'
echo ' -u: Specify the UID of the user for which the redirection is not'
echo ' applied. Typically, this is the UID of the proxy container'
# shellcheck disable=SC2016
echo ' (default to uid of $ENVOY_USER, uid of istio_proxy, or 1337)'
echo ' -g: Specify the GID of the user for which the redirection is not'
echo ' applied. (same default value as -u param)'
echo ' -m: The mode used to redirect inbound connections to Envoy, either "REDIRECT" or "TPROXY"'
# shellcheck disable=SC2016
echo ' (default to $ISTIO_INBOUND_INTERCEPTION_MODE)'
echo ' -b: Comma separated list of inbound ports for which traffic is to be redirected to Envoy (optional). The'
echo ' wildcard character "*" can be used to configure redirection for all ports. An empty list will disable'
# shellcheck disable=SC2016
echo ' all inbound redirection (default to $ISTIO_INBOUND_PORTS)'
echo ' -d: Comma separated list of inbound ports to be excluded from redirection to Envoy (optional). Only applies'
# shellcheck disable=SC2016
echo ' when all inbound traffic (i.e. "*") is being redirected (default to $ISTIO_LOCAL_EXCLUDE_PORTS)'
echo ' -i: Comma separated list of IP ranges in CIDR form to redirect to envoy (optional). The wildcard'
echo ' character "*" can be used to redirect all outbound traffic. An empty list will disable all outbound'
# shellcheck disable=SC2016
echo ' redirection (default to $ISTIO_SERVICE_CIDR)'
echo ' -x: Comma separated list of IP ranges in CIDR form to be excluded from redirection. Only applies when all '
# shellcheck disable=SC2016
echo ' outbound traffic (i.e. "*") is being redirected (default to $ISTIO_SERVICE_EXCLUDE_CIDR).'
echo ' -o: Comma separated list of outbound ports to be excluded from redirection to Envoy (optional).'
echo ' -k: Comma separated list of virtual interfaces whose inbound traffic (from VM)'
echo ' will be treated as outbound (optional)'
echo ' -t: Unit testing, only functions are loaded and no other instructions are executed.'
echo ' -h: Displays usage information and exits.'
# shellcheck disable=SC2016
echo ''
}
那这条命令到底执行了一些什么,我们继续来看一下接下来的代码
§ 启动参数转换为对应变量
第一步是把各个参数转换成相应的变量,或者直接获取环境变量
如果没有指定启动参数会用 /var/lib/istio/envoy/sidecar.env 中的环境变量
如果既没有指定启动参数,sidecar.env文件中又没有配置环境变量,则用默认的参数
以下代码说明了参数和变量的转换关系:
while getopts ":p:z:u:g:m:b:d:o:i:x:k:ht" opt; do
case ${opt} in
p)
PROXY_PORT=${OPTARG}
;;
z)
PROXY_INBOUND_CAPTURE_PORT=${OPTARG}
;;
u)
PROXY_UID=${OPTARG}
;;
g)
PROXY_GID=${OPTARG}
;;
m)
INBOUND_INTERCEPTION_MODE=${OPTARG}
;;
b)
INBOUND_PORTS_INCLUDE=${OPTARG}
;;
d)
INBOUND_PORTS_EXCLUDE=${OPTARG}
;;
i)
OUTBOUND_IP_RANGES_INCLUDE=${OPTARG}
;;
x)
OUTBOUND_IP_RANGES_EXCLUDE=${OPTARG}
;;
o)
OUTBOUND_PORTS_EXCLUDE=${OPTARG}
;;
k)
KUBEVIRT_INTERFACES=${OPTARG}
;;
t)
echo "Unit testing is specified..."
return
;;
h)
usage
exit 0
;;
\?)
echo "Invalid option: -$OPTARG" >&2
usage
exit 1
;;
esac
done
根据以上转换关系,我们继续以istio init容器启动命令来看
istio-iptables -p 15001 -z 15006 -u 1337 -m REDIRECT -i * -x -b * -d 15020
可以得到下面的转换关系
-p PROXY_PORT 15001
-z PROXY_INBOUND_CAPTURE_PORT 15006
-b INBOUND_PORTS_INCLUD *
-m INBOUND_INTERCEPTION_MODE REDIRECT
-d INBOUND_PORTS_INCLUDE 15020
§ 执行iptables命令生成iptables规则
弄清楚环境变量的取值之后来看,真正生成iptables规则的代码
生成ISTIO_REDIRECT链
# 生成ISTIO_REDIRECT链
iptables -t nat -N ISTIO_REDIRECT
# 在ISTIO_REDIRECT链中追加一条REDIRECT规则,重定向到Envoy的Outbound端口(15001)
iptables -t nat -A ISTIO_REDIRECT -p tcp -j REDIRECT --to-port "${PROXY_PORT}"
生成ISTIO_IN_REDIRECT链
# 生成 ISTIO_REDIRECT链
iptables -t nat -N ISTIO_IN_REDIRECT
if [ "${INBOUND_PORTS_INCLUDE}" == "*" ]; then
# 因为启动参数设置了-b即INBOUND_PORTS_INCLUD为*所以进入这个分支
# 在ISTIO_IN_REDIRECT链中追加一条REDIRECT规则,重定向到Envoy的Inbound端口(15006)
iptables -t nat -A ISTIO_IN_REDIRECT -p tcp -j REDIRECT --to-port "${PROXY_INBOUND_CAPTURE_PORT}"
else
iptables -t nat -A ISTIO_IN_REDIRECT -p tcp -j REDIRECT --to-port "${PROXY_PORT}"
fi
生成ISTIO_INBOUND链
if [ -n "${INBOUND_PORTS_INCLUDE}" ]; then
if [ "${INBOUND_INTERCEPTION_MODE}" = "TPROXY" ] ; then
...
else
table=nat
fi
# 启动参数-m为REDIRECT所以进入这个分支
iptables -t ${table} -N ISTIO_INBOUND
iptables -t ${table} -A PREROUTING -p tcp -j ISTIO_INBOUND
# 新增规则RETURN,保证一些端口不进行重定向,例如ssh端口22,Envoy健康检查端口
if [ "${INBOUND_PORTS_INCLUDE}" == "*" ]; then
# Makes sure SSH is not redirected
iptables -t ${table} -A ISTIO_INBOUND -p tcp --dport 22 -j RETURN
# Apply any user-specified port exclusions.
if [ -n "${INBOUND_PORTS_EXCLUDE}" ]; then
for port in ${INBOUND_PORTS_EXCLUDE}; do
iptables -t ${table} -A ISTIO_INBOUND -p tcp --dport "${port}" -j RETURN
done
fi
# Redirect remaining inbound traffic to Envoy.
if [ "${INBOUND_INTERCEPTION_MODE}" = "TPROXY" ]; then
...
else
iptables -t nat -A ISTIO_INBOUND -p tcp -j ISTIO_IN_REDIRECT
fi
else
for port in ${INBOUND_PORTS_INCLUDE}; do
if [ "${INBOUND_INTERCEPTION_MODE}" = "TPROXY" ]; then
...
else
iptables -t nat -A ISTIO_INBOUND -p tcp --dport "${port}" -j ISTIO_IN_REDIRECT
fi
done
fi
fi
生成ISTIO_OUTPUT链
# Create a new chain for selectively redirecting outbound packets to Envoy.
iptables -t nat -N ISTIO_OUTPUT
# Jump to the ISTIO_OUTPUT chain from OUTPUT chain for all tcp traffic.
iptables -t nat -A OUTPUT -p tcp -j ISTIO_OUTPUT
# Apply port based exclusions. Must be applied before connections back to self
# are redirected.
if [ -n "${OUTBOUND_PORTS_EXCLUDE}" ]; then
for port in ${OUTBOUND_PORTS_EXCLUDE}; do
iptables -t nat -A ISTIO_OUTPUT -p tcp --dport "${port}" -j RETURN
done
fi
# 127.0.0.6 is bind connect from inbound passthrough cluster
iptables -t nat -A ISTIO_OUTPUT -o lo -s 127.0.0.6/32 -j RETURN
if [ -z "${DISABLE_REDIRECTION_ON_LOCAL_LOOPBACK-}" ]; then
# Redirect app calls back to itself via Envoy when using the service VIP or endpoint
# address, e.g. appN => Envoy (client) => Envoy (server) => appN.
iptables -t nat -A ISTIO_OUTPUT -o lo ! -d 127.0.0.1/32 -j ISTIO_IN_REDIRECT
fi
for uid in ${PROXY_UID}; do
# Avoid infinite loops. Don't redirect Envoy traffic directly back to
# Envoy for non-loopback traffic.
iptables -t nat -A ISTIO_OUTPUT -m owner --uid-owner "${uid}" -j RETURN
done
for gid in ${PROXY_GID}; do
# Avoid infinite loops. Don't redirect Envoy traffic directly back to
# Envoy for non-loopback traffic.
iptables -t nat -A ISTIO_OUTPUT -m owner --gid-owner "${gid}" -j RETURN
done
# Skip redirection for Envoy-aware applications and
# container-to-container traffic both of which explicitly use
# localhost.
iptables -t nat -A ISTIO_OUTPUT -d 127.0.0.1/32 -j RETURN
# Apply outbound IPv4 exclusions. Must be applied before inclusions.
if [ ${#ipv4_ranges_exclude[@]} -gt 0 ]; then
for cidr in "${ipv4_ranges_exclude[@]}"; do
iptables -t nat -A ISTIO_OUTPUT -d "${cidr}" -j RETURN
执行iptables save保存上面生成的所有规则
最后一个dump函数保存上面创建的所有链
function dump {
iptables-save
ip6tables-save
}
最终执行完整个脚本之后可以看到生成的iptables如下
§ 执行命令之后生成的iptables规则有
$ nsenter -t 20533 -n iptables -t nat -S
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-N ISTIO_INBOUND
-N ISTIO_IN_REDIRECT
-N ISTIO_OUTPUT
-N ISTIO_REDIRECT
-A PREROUTING -p tcp -j ISTIO_INBOUND
-A OUTPUT -p tcp -j ISTIO_OUTPUT
-A ISTIO_INBOUND -p tcp -m tcp --dport 22 -j RETURN
-A ISTIO_INBOUND -p tcp -m tcp --dport 15020 -j RETURN
-A ISTIO_INBOUND -p tcp -j ISTIO_IN_REDIRECT
-A ISTIO_IN_REDIRECT -p tcp -j REDIRECT --to-ports 15006
-A ISTIO_OUTPUT -s 127.0.0.6/32 -o lo -j RETURN
-A ISTIO_OUTPUT ! -d 127.0.0.1/32 -o lo -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -m owner --uid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -m owner --gid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -d 127.0.0.1/32 -j RETURN
-A ISTIO_OUTPUT -j ISTIO_REDIRECT
-A ISTIO_REDIRECT -p tcp -j REDIRECT --to-ports 15001
详细的关系如下:
nsenter -t 20533 -n iptables -t nat -L
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
ISTIO_INBOUND tcp -- anywhere anywhere
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
ISTIO_OUTPUT tcp -- anywhere anywhere
Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
Chain ISTIO_INBOUND (1 references)
target prot opt source destination
RETURN tcp -- anywhere anywhere tcp dpt:ssh
RETURN tcp -- anywhere anywhere tcp dpt:15020
ISTIO_IN_REDIRECT tcp -- anywhere anywhere
Chain ISTIO_IN_REDIRECT (2 references)
target prot opt source destination
REDIRECT tcp -- anywhere anywhere redir ports 15006
Chain ISTIO_OUTPUT (1 references)
target prot opt source destination
RETURN all -- 127.0.0.6 anywhere
ISTIO_IN_REDIRECT all -- anywhere !localhost
RETURN all -- anywhere anywhere owner UID match 1337
RETURN all -- anywhere anywhere owner GID match 1337
RETURN all -- anywhere localhost
ISTIO_REDIRECT all -- anywhere anywhere
Chain ISTIO_REDIRECT (1 references)
target prot opt source destination
REDIRECT tcp -- anywhere anywhere redir ports 15001
关于生成的iptables具体意思和怎么工作的,有挺多文章写的很详细,可以参考附录中的一些链接具体了解
关于iptables把流量重定向到Envoy之后,Envoy怎么处理的参考:
https://www.servicemesher.com/blog/envoy-sidecar-routing-of-istio-service-mesh-deep-dive/
§ 附录
§ 如何查看容器中的istio生成的iptables
注意:iptables直接执行iptables命令可能看不到信息,这是由于每个容器有自己的netns,需要进入他自己的netns才能看到,可以使用以下几条命令进入:
# step 1: 根据docker id找到pid
$ docker inspect d6fae8dbf00f|grep Pid
"Pid": 20533,
"PidMode": "",
"PidsLimit": 0,
# step 2: 根据Pid 进入到netns查看iptables
$ nsenter -t 20533 -n iptables -t nat -S
§ 了解更多iptables命令
--append -A chain Append to chain
--check -C chain Check for the existence of a rule
--delete -D chain Delete matching rule from chain
--delete -D chain rulenum
Delete rule rulenum (1 = first) from chain
--insert -I chain [rulenum]
Insert in chain as rulenum (default 1=first)
--replace -R chain rulenum
Replace rule rulenum (1 = first) in chain
--list -L [chain [rulenum]]
List the rules in a chain or all chains
--list-rules -S [chain [rulenum]]
Print the rules in a chain or all chains
--flush -F [chain] Delete all rules in chain or all chains
--zero -Z [chain [rulenum]]
Zero counters in chain or all chains
--new -N chain Create a new user-defined chain
--delete-chain
-X [chain] Delete a user-defined chain
--policy -P chain target
Change policy on chain to target
--rename-chain
-E old-chain new-chain
Change chain name, (moving any references)
http://www.zsythink.net/archives/1517
https://wangchujiang.com/linux-command/c/iptables.html
§ 其他参考
https://jimmysong.io/istio-handbook/concepts/sidecar-injection-deep-dive.html
- 点赞
- 收藏
- 关注作者
评论(0)