istio sidecar中创建iptables流程
【摘要】 § istio-iptables.sh 源码解析§ 启动参数解析代码istio的init container中初始化iptables的命令如下： istio-iptables -p 15001 -z 15006 -u 1337 -m REDIRECT -i * -x -b * -d 15020这条命令的意思是什么呢？以下是istio-iptables.sh中的usage函数，表明...
§ istio-iptables.sh 源码解析
§ 启动参数解析代码
istio的init container中初始化iptables的命令如下：
istio-iptables -p 15001 -z 15006 -u 1337 -m REDIRECT -i * -x -b * -d 15020
这条命令的意思是什么呢？
以下是istio-iptables.sh中的usage函数，表明了每个参数一意思以及默认值，看完这个就可以很清楚的明白这条命令的意思
其实就是让Envoy代理拦截所有进出Pod的流量，将入站流量重定向到Envoy的15006端口（Inbound端口），将出站流量重定向到15001端口（Outbound端口），同时排除15020端口不进行重定向（健康检查端口）
function usage() {
echo "${0} -p PORT -u UID -g GID [-m mode] [-b ports] [-d ports] [-i CIDR] [-x CIDR] [-k interfaces] [-t] [-h]"
echo ''
# shellcheck disable=SC2016
echo ' -p: Specify the envoy port to which redirect all TCP traffic (default $ENVOY_PORT = 15001)'
# shellcheck disable=SC2016
echo ' -z: Port to which all inbound TCP traffic to the pod/VM should be redirected to. For REDIRECT only (default $INBOUND_CAPTURE_PORT = 15006)'
echo ' -u: Specify the UID of the user for which the redirection is not'
echo ' applied. Typically, this is the UID of the proxy container'
# shellcheck disable=SC2016
echo ' (default to uid of $ENVOY_USER, uid of istio_proxy, or 1337)'
echo ' -g: Specify the GID of the user for which the redirection is not'
echo ' applied. (same default value as -u param)'
echo ' -m: The mode used to redirect inbound connections to Envoy, either "REDIRECT" or "TPROXY"'
# shellcheck disable=SC2016
echo ' (default to $ISTIO_INBOUND_INTERCEPTION_MODE)'
echo ' -b: Comma separated list of inbound ports for which traffic is to be redirected to Envoy (optional). The'
echo ' wildcard character "*" can be used to configure redirection for all ports. An empty list will disable'
# shellcheck disable=SC2016
echo ' all inbound redirection (default to $ISTIO_INBOUND_PORTS)'
echo ' -d: Comma separated list of inbound ports to be excluded from redirection to Envoy (optional). Only applies'
# shellcheck disable=SC2016
echo ' when all inbound traffic (i.e. "*") is being redirected (default to $ISTIO_LOCAL_EXCLUDE_PORTS)'
echo ' -i: Comma separated list of IP ranges in CIDR form to redirect to envoy (optional). The wildcard'
echo ' character "*" can be used to redirect all outbound traffic. An empty list will disable all outbound'
# shellcheck disable=SC2016
echo ' redirection (default to $ISTIO_SERVICE_CIDR)'
echo ' -x: Comma separated list of IP ranges in CIDR form to be excluded from redirection. Only applies when all '
# shellcheck disable=SC2016
echo ' outbound traffic (i.e. "*") is being redirected (default to $ISTIO_SERVICE_EXCLUDE_CIDR).'
echo ' -o: Comma separated list of outbound ports to be excluded from redirection to Envoy (optional).'
echo ' -k: Comma separated list of virtual interfaces whose inbound traffic (from VM)'
echo ' will be treated as outbound (optional)'
echo ' -t: Unit testing, only functions are loaded and no other instructions are executed.'
echo ' -h: Displays usage information and exits.'
# shellcheck disable=SC2016
echo ''
}那这条命令到底执行了一些什么，我们继续来看一下接下来的代码
§ 启动参数转换为对应变量
第一步是把各个参数转换成相应的变量，或者直接获取环境变量
如果没有指定启动参数会用 /var/lib/istio/envoy/sidecar.env 中的环境变量
如果既没有指定启动参数，sidecar.env文件中又没有配置环境变量，则用默认的参数
以下代码说明了参数和变量的转换关系：
while getopts ":p:z:u:g:m:b:d:o:i:x:k:ht" opt; do
case ${opt} in
p)
PROXY_PORT=${OPTARG}
;;
z)
PROXY_INBOUND_CAPTURE_PORT=${OPTARG}
;;
u)
PROXY_UID=${OPTARG}
;;
g)
PROXY_GID=${OPTARG}
;;
m)
INBOUND_INTERCEPTION_MODE=${OPTARG}
;;
b)
INBOUND_PORTS_INCLUDE=${OPTARG}
;;
d)
INBOUND_PORTS_EXCLUDE=${OPTARG}
;;
i)
OUTBOUND_IP_RANGES_INCLUDE=${OPTARG}
;;
x)
OUTBOUND_IP_RANGES_EXCLUDE=${OPTARG}
;;
o)
OUTBOUND_PORTS_EXCLUDE=${OPTARG}
;;
k)
KUBEVIRT_INTERFACES=${OPTARG}
;;
t)
echo "Unit testing is specified..."
return
;;
h)
usage
exit 0
;;
\?)
echo "Invalid option: -$OPTARG" >&2
usage
exit 1
;;
esac
done根据以上转换关系，我们继续以istio init容器启动命令来看
istio-iptables -p 15001 -z 15006 -u 1337 -m REDIRECT -i * -x -b * -d 15020
可以得到下面的转换关系
-p PROXY_PORT 15001 -z PROXY_INBOUND_CAPTURE_PORT 15006 -b INBOUND_PORTS_INCLUD * -m INBOUND_INTERCEPTION_MODE REDIRECT -d INBOUND_PORTS_INCLUDE 15020
§ 执行iptables命令生成iptables规则
弄清楚环境变量的取值之后来看，真正生成iptables规则的代码
生成ISTIO_REDIRECT链
# 生成ISTIO_REDIRECT链
iptables -t nat -N ISTIO_REDIRECT
# 在ISTIO_REDIRECT链中追加一条REDIRECT规则，重定向到Envoy的Outbound端口（15001）
iptables -t nat -A ISTIO_REDIRECT -p tcp -j REDIRECT --to-port "${PROXY_PORT}"生成ISTIO_IN_REDIRECT链
# 生成 ISTIO_REDIRECT链
iptables -t nat -N ISTIO_IN_REDIRECT
if [ "${INBOUND_PORTS_INCLUDE}" == "*" ]; then
# 因为启动参数设置了-b即INBOUND_PORTS_INCLUD为*所以进入这个分支
# 在ISTIO_IN_REDIRECT链中追加一条REDIRECT规则，重定向到Envoy的Inbound端口（15006）
iptables -t nat -A ISTIO_IN_REDIRECT -p tcp -j REDIRECT --to-port "${PROXY_INBOUND_CAPTURE_PORT}"
else
iptables -t nat -A ISTIO_IN_REDIRECT -p tcp -j REDIRECT --to-port "${PROXY_PORT}"
fi生成ISTIO_INBOUND链
if [ -n "${INBOUND_PORTS_INCLUDE}" ]; then
if [ "${INBOUND_INTERCEPTION_MODE}" = "TPROXY" ] ; then
...
else
table=nat
fi
# 启动参数-m为REDIRECT所以进入这个分支
iptables -t ${table} -N ISTIO_INBOUND
iptables -t ${table} -A PREROUTING -p tcp -j ISTIO_INBOUND
# 新增规则RETURN，保证一些端口不进行重定向，例如ssh端口22，Envoy健康检查端口
if [ "${INBOUND_PORTS_INCLUDE}" == "*" ]; then
# Makes sure SSH is not redirected
iptables -t ${table} -A ISTIO_INBOUND -p tcp --dport 22 -j RETURN
# Apply any user-specified port exclusions.
if [ -n "${INBOUND_PORTS_EXCLUDE}" ]; then
for port in ${INBOUND_PORTS_EXCLUDE}; do
iptables -t ${table} -A ISTIO_INBOUND -p tcp --dport "${port}" -j RETURN
done
fi
# Redirect remaining inbound traffic to Envoy.
if [ "${INBOUND_INTERCEPTION_MODE}" = "TPROXY" ]; then
...
else
iptables -t nat -A ISTIO_INBOUND -p tcp -j ISTIO_IN_REDIRECT
fi
else
for port in ${INBOUND_PORTS_INCLUDE}; do
if [ "${INBOUND_INTERCEPTION_MODE}" = "TPROXY" ]; then
...
else
iptables -t nat -A ISTIO_INBOUND -p tcp --dport "${port}" -j ISTIO_IN_REDIRECT
fi
done
fi
fi生成ISTIO_OUTPUT链
# Create a new chain for selectively redirecting outbound packets to Envoy.
iptables -t nat -N ISTIO_OUTPUT
# Jump to the ISTIO_OUTPUT chain from OUTPUT chain for all tcp traffic.
iptables -t nat -A OUTPUT -p tcp -j ISTIO_OUTPUT
# Apply port based exclusions. Must be applied before connections back to self
# are redirected.
if [ -n "${OUTBOUND_PORTS_EXCLUDE}" ]; then
for port in ${OUTBOUND_PORTS_EXCLUDE}; do
iptables -t nat -A ISTIO_OUTPUT -p tcp --dport "${port}" -j RETURN
done
fi
# 127.0.0.6 is bind connect from inbound passthrough cluster
iptables -t nat -A ISTIO_OUTPUT -o lo -s 127.0.0.6/32 -j RETURN
if [ -z "${DISABLE_REDIRECTION_ON_LOCAL_LOOPBACK-}" ]; then
# Redirect app calls back to itself via Envoy when using the service VIP or endpoint
# address, e.g. appN => Envoy (client) => Envoy (server) => appN.
iptables -t nat -A ISTIO_OUTPUT -o lo ! -d 127.0.0.1/32 -j ISTIO_IN_REDIRECT
fi
for uid in ${PROXY_UID}; do
# Avoid infinite loops. Don't redirect Envoy traffic directly back to
# Envoy for non-loopback traffic.
iptables -t nat -A ISTIO_OUTPUT -m owner --uid-owner "${uid}" -j RETURN
done
for gid in ${PROXY_GID}; do
# Avoid infinite loops. Don't redirect Envoy traffic directly back to
# Envoy for non-loopback traffic.
iptables -t nat -A ISTIO_OUTPUT -m owner --gid-owner "${gid}" -j RETURN
done
# Skip redirection for Envoy-aware applications and
# container-to-container traffic both of which explicitly use
# localhost.
iptables -t nat -A ISTIO_OUTPUT -d 127.0.0.1/32 -j RETURN
# Apply outbound IPv4 exclusions. Must be applied before inclusions.
if [ ${#ipv4_ranges_exclude[@]} -gt 0 ]; then
for cidr in "${ipv4_ranges_exclude[@]}"; do
iptables -t nat -A ISTIO_OUTPUT -d "${cidr}" -j RETURN执行iptables save保存上面生成的所有规则
最后一个dump函数保存上面创建的所有链
function dump {
iptables-save
ip6tables-save
}最终执行完整个脚本之后可以看到生成的iptables如下
§ 执行命令之后生成的iptables规则有
$ nsenter -t 20533 -n iptables -t nat -S -P PREROUTING ACCEPT -P INPUT ACCEPT -P OUTPUT ACCEPT -P POSTROUTING ACCEPT -N ISTIO_INBOUND -N ISTIO_IN_REDIRECT -N ISTIO_OUTPUT -N ISTIO_REDIRECT -A PREROUTING -p tcp -j ISTIO_INBOUND -A OUTPUT -p tcp -j ISTIO_OUTPUT -A ISTIO_INBOUND -p tcp -m tcp --dport 22 -j RETURN -A ISTIO_INBOUND -p tcp -m tcp --dport 15020 -j RETURN -A ISTIO_INBOUND -p tcp -j ISTIO_IN_REDIRECT -A ISTIO_IN_REDIRECT -p tcp -j REDIRECT --to-ports 15006 -A ISTIO_OUTPUT -s 127.0.0.6/32 -o lo -j RETURN -A ISTIO_OUTPUT ! -d 127.0.0.1/32 -o lo -j ISTIO_IN_REDIRECT -A ISTIO_OUTPUT -m owner --uid-owner 1337 -j RETURN -A ISTIO_OUTPUT -m owner --gid-owner 1337 -j RETURN -A ISTIO_OUTPUT -d 127.0.0.1/32 -j RETURN -A ISTIO_OUTPUT -j ISTIO_REDIRECT -A ISTIO_REDIRECT -p tcp -j REDIRECT --to-ports 15001
详细的关系如下：
nsenter -t 20533 -n iptables -t nat -L Chain PREROUTING (policy ACCEPT) target prot opt source destination ISTIO_INBOUND tcp -- anywhere anywhere Chain INPUT (policy ACCEPT) target prot opt source destination Chain OUTPUT (policy ACCEPT) target prot opt source destination ISTIO_OUTPUT tcp -- anywhere anywhere Chain POSTROUTING (policy ACCEPT) target prot opt source destination Chain ISTIO_INBOUND (1 references) target prot opt source destination RETURN tcp -- anywhere anywhere tcp dpt:ssh RETURN tcp -- anywhere anywhere tcp dpt:15020 ISTIO_IN_REDIRECT tcp -- anywhere anywhere Chain ISTIO_IN_REDIRECT (2 references) target prot opt source destination REDIRECT tcp -- anywhere anywhere redir ports 15006 Chain ISTIO_OUTPUT (1 references) target prot opt source destination RETURN all -- 127.0.0.6 anywhere ISTIO_IN_REDIRECT all -- anywhere !localhost RETURN all -- anywhere anywhere owner UID match 1337 RETURN all -- anywhere anywhere owner GID match 1337 RETURN all -- anywhere localhost ISTIO_REDIRECT all -- anywhere anywhere Chain ISTIO_REDIRECT (1 references) target prot opt source destination REDIRECT tcp -- anywhere anywhere redir ports 15001
关于生成的iptables具体意思和怎么工作的，有挺多文章写的很详细，可以参考附录中的一些链接具体了解
关于iptables把流量重定向到Envoy之后，Envoy怎么处理的参考：
https://www.servicemesher.com/blog/envoy-sidecar-routing-of-istio-service-mesh-deep-dive/
§ 附录
§ 如何查看容器中的istio生成的iptables
注意：iptables直接执行iptables命令可能看不到信息，这是由于每个容器有自己的netns，需要进入他自己的netns才能看到，可以使用以下几条命令进入：
# step 1: 根据docker id找到pid $ docker inspect d6fae8dbf00f|grep Pid "Pid": 20533, "PidMode": "", "PidsLimit": 0, # step 2: 根据Pid 进入到netns查看iptables $ nsenter -t 20533 -n iptables -t nat -S
§ 了解更多iptables命令
--append -A chain Append to chain --check -C chain Check for the existence of a rule --delete -D chain Delete matching rule from chain --delete -D chain rulenum Delete rule rulenum (1 = first) from chain --insert -I chain [rulenum] Insert in chain as rulenum (default 1=first) --replace -R chain rulenum Replace rule rulenum (1 = first) in chain --list -L [chain [rulenum]] List the rules in a chain or all chains --list-rules -S [chain [rulenum]] Print the rules in a chain or all chains --flush -F [chain] Delete all rules in chain or all chains --zero -Z [chain [rulenum]] Zero counters in chain or all chains --new -N chain Create a new user-defined chain --delete-chain -X [chain] Delete a user-defined chain --policy -P chain target Change policy on chain to target --rename-chain -E old-chain new-chain Change chain name, (moving any references)
http://www.zsythink.net/archives/1517
https://wangchujiang.com/linux-command/c/iptables.html
§ 其他参考
https://jimmysong.io/istio-handbook/concepts/sidecar-injection-deep-dive.html
【版权声明】本文为华为云社区用户原创内容，转载时必须标注文章的来源（华为云社区）、文章链接、文章作者等基本信息， 否则作者和本社区有权追究责任。如果您发现本社区中有涉嫌抄袭的内容，欢迎发送邮件进行举报，并提供相关证据，一经查实，本社区将立刻删除涉嫌侵权内容，举报邮箱：
cloudbbs@huaweicloud.com
- 点赞
- 收藏
- 关注作者
评论（0）