本地访问失败

C:\Windows\System32>curl 47.88.85.25:30080
curl: (7) Failed to connect to 47.88.85.25 port 30080 after 3206 ms: Could not connect to server

pod 正常

root@iZrj9g4hb568m46labgv2eZ:~/k8s-cfg# kubectl get pod -n cicd -o wide
NAME                      READY   STATUS    RESTARTS   AGE    IP               NODE                      NOMINATED NODE   READINESS GATES
gitlab-68cdfc9969-2csxl   1/1     Running   0          100m   192.168.192.20   izrj943cxeelu8yy8nt47sz   <none>           <none>

确认安全组规则入规则、tcp、端口已放开,抓包验证(k8s 节点执行 tcpdump、本地客户端执行 curl):

root@iZrj9g4hb568m46labgv2eZ:~/k8s-cfg# sudo tcpdump -i any port 30080 -n
tcpdump: data link type LINUX_SLL2
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
00:30:27.119344 eth0  In  IP 223.160.129.161.48688 > 10.0.1.245.30080: Flags [S], seq 3025758360, win 65535, options [mss 1380,nop,wscale 8,nop,nop,sackOK], length 0
00:30:27.119626 eth0  Out IP 10.0.1.245.30080 > 223.160.129.161.48688: Flags [R.], seq 0, ack 3025758361, win 0, length 0
00:30:27.842769 eth0  In  IP 223.160.129.161.48689 > 10.0.1.245.30080: Flags [S], seq 3025758360, win 65535, options [mss 1380,nop,wscale 8,nop,nop,sackOK], length 0
00:30:27.843001 eth0  Out IP 10.0.1.245.30080 > 223.160.129.161.48689: Flags [R.], seq 0, ack 3025758361, win 0, length 0
00:30:28.532598 eth0  In  IP 223.160.129.161.48690 > 10.0.1.245.30080: Flags [S], seq 3025758360, win 65535, options [mss 1380,nop,wscale 8,nop,nop,sackOK], length 0
00:30:28.532821 eth0  Out IP 10.0.1.245.30080 > 223.160.129.161.48690: Flags [R.], seq 0, ack 3025758361, win 0, length 0
00:30:29.282885 eth0  In  IP 223.160.129.161.48691 > 10.0.1.245.30080: Flags [S], seq 3025758360, win 65535, options [mss 1380,nop,wscale 8,nop,nop,sackOK], length 0
00:30:29.283108 eth0  Out IP 10.0.1.245.30080 > 223.160.129.161.48691: Flags [R.], seq 0, ack 3025758361, win 0, length 0
00:30:30.142524 eth0  In  IP 223.160.129.161.48692 > 10.0.1.245.30080: Flags [S], seq 3025758360, win 65535, options [mss 1380,nop,wscale 8,nop,nop,sackOK], length 0
00:30:30.142738 eth0  Out IP 10.0.1.245.30080 > 223.160.129.161.48692: Flags [R.], seq 0, ack 3025758361, win 0, length 0

外部 IP 223.160.129.161 正在尝试连接服务器 10.0.1.245 的 30080 端口,但每一次 TCP 握手请求([S],SYN)都被服务器以 [R.](RST+ACK) 拒绝了。
30080 端口上没有服务在监听,或者防火墙直接拒绝了连接。

检查本地防火墙,根本没有启动

root@iZrj9g4hb568m46labgv2eZ:~/k8s-cfg# sudo ufw status
Status: inactive

k8s 节点本地访问失败

root@iZrj9g4hb568m46labgv2eZ:~/k8s-cfg# curl localhost:30080
curl: (7) Failed to connect to localhost port 30080 after 0 ms: Couldn't connect to server

k8s 节点本地私有ip访问失败

root@iZrj9g4hb568m46labgv2eZ:~/k8s-cfg# curl 10.0.1.245:30080
curl: (7) Failed to connect to 10.0.1.245 port 30080 after 0 ms: Couldn't connect to server

分析 kube-proxy

root@iZrj9g4hb568m46labgv2eZ:~/k8s-cfg# kubectl get cm -n kube-system kube-proxy -o yaml | grep mode
    mode: "" # iptables
root@iZrj9g4hb568m46labgv2eZ:~/k8s-cfg# sudo iptables -t nat -L KUBE-NODEPORTS | grep 30080
KUBE-EXT-PXXAYWTNW4C7MHDB  tcp  --  anywhere             localhost/8          /* cicd/gitlab:http */ tcp dpt:30080 nfacct-name  localhost_nps_accepted_pkts
KUBE-EXT-PXXAYWTNW4C7MHDB  tcp  --  anywhere             anywhere             /* cicd/gitlab:http */ tcp dpt:30080
root@iZrj9g4hb568m46labgv2eZ:~/k8s-cfg# sudo iptables -t nat -L KUBE-EXT-PXXAYWTNW4C7MHDB -n -v
Chain KUBE-EXT-PXXAYWTNW4C7MHDB (2 references)
 pkts bytes target     prot opt in     out     source               destination         
   79  4624 KUBE-MARK-MASQ  0    --  *      *       0.0.0.0/0            0.0.0.0/0            /* masquerade traffic for cicd/gitlab:http external destinations */
   79  4624 KUBE-SVC-PXXAYWTNW4C7MHDB  0    --  *      *       0.0.0.0/0            0.0.0.0/0
root@iZrj9g4hb568m46labgv2eZ:~/k8s-cfg# sudo iptables -t nat -L KUBE-SVC-PXXAYWTNW4C7MHDB -n -v
Chain KUBE-SVC-PXXAYWTNW4C7MHDB (2 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 KUBE-SEP-UKXDEMYDPMYPF2Z6  0    --  *      *       0.0.0.0/0            0.0.0.0/0            /* cicd/gitlab:http -> 192.168.192.20:8080 */

再次抓包(k8s 节点执行 tcpdump、本地客户端执行 curl)

root@iZrj9g4hb568m46labgv2eZ:~/k8s-cfg# sudo tcpdump -i any host 192.168.192.20 and port 8080 -n
tcpdump: data link type LINUX_SLL2
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
00:46:26.863753 tunl0 Out IP 192.168.82.64.9839 > 192.168.192.20.8080: Flags [S], seq 3746324047, win 65535, options [mss 1380,nop,wscale 8,nop,nop,sackOK], length 0
00:46:26.864033 tunl0 In  IP 192.168.192.20.8080 > 192.168.82.64.9839: Flags [R.], seq 0, ack 3746324048, win 0, length 0
00:46:27.583588 tunl0 Out IP 192.168.82.64.18530 > 192.168.192.20.8080: Flags [S], seq 3746324047, win 65535, options [mss 1380,nop,wscale 8,nop,nop,sackOK], length 0
00:46:27.583792 tunl0 In  IP 192.168.192.20.8080 > 192.168.82.64.18530: Flags [R.], seq 0, ack 3746324048, win 0, length 0
00:46:28.303083 tunl0 Out IP 192.168.82.64.56700 > 192.168.192.20.8080: Flags [S], seq 3746324047, win 65535, options [mss 1380,nop,wscale 8,nop,nop,sackOK], length 0
00:46:28.303298 tunl0 In  IP 192.168.192.20.8080 > 192.168.82.64.56700: Flags [R.], seq 0, ack 3746324048, win 0, length 0
00:46:29.004171 tunl0 Out IP 192.168.82.64.20565 > 192.168.192.20.8080: Flags [S], seq 3746324047, win 65535, options [mss 1380,nop,wscale 8,nop,nop,sackOK], length 0
00:46:29.004376 tunl0 In  IP 192.168.192.20.8080 > 192.168.82.64.20565: Flags [R.], seq 0, ack 3746324048, win 0, length 0
00:46:29.861780 tunl0 Out IP 192.168.82.64.17245 > 192.168.192.20.8080: Flags [S], seq 3746324047, win 65535, options [mss 1380,nop,wscale 8,nop,nop,sackOK], length 0
00:46:29.861969 tunl0 In  IP 192.168.192.20.8080 > 192.168.82.64.17245: Flags [R.], seq 0, ack 3746324048, win 0, length 0

SYN 已经到达 Pod 网卡(192.168.192.20:8080)
Pod 里没人监听 8080,所以内核立刻回 RST(win 0)

看看 pod 内 8080 监听情况

root@iZrj9g4hb568m46labgv2eZ:~/k8s-cfg# kubectl exec -n cicd gitlab-68cdfc9969-2csxl -it -- \
  sh -c '
    ss -ltn 2>/dev/null | grep ":8080" ||
    netstat -ltn 2>/dev/null | grep ":8080" ||
    awk '\''$2=="00000000:1F90"'\'' /proc/net/tcp ||
    echo "8080 not listening"
  '
Defaulted container "gitlab" out of: gitlab, fix-perm (init)
tcp        0      0 127.0.0.1:8080          0.0.0.0:*               LISTEN

好的,问题找到了,是在 127.0.0.1 进行监听的,而流量来自 192.168.192.20

GitLab 是用 Ruby on Rails 开发的,它的 Web 层就是由 Puma 提供的。
在 GitLab 中(如 Kubernetes 或 Omnibus 安装):Puma 在 TCP 地址 127.0.0.1:8080 上监听。
https://gitlab.cn/docs/omnibus/troubleshooting/#tcp-ports-for-gitlab-services-are-already-taken
解决方式是将 puma[‘listen’] 设置为 0.0.0.0

Logo

码道开发者社区,聚焦华为云码道 CodeArts 代码智能体,沉淀 Agent、Skill、鸿蒙开发实战内容,供开发者查阅资料、交流技术、分享工程实践

更多推荐