- 主题:关于在k8s平台实现fail2ban的尝试
在k8s上装了一个gitlab接口，用ingress-nginx暴露了2222端口作为ssh服务端口，供远
程提交代码。这是前言
后来就在gitlab日志里发现n多尝试登录的信息，基本上每分钟都有好几次，这不行啊，
有洁癖的我受不了。
在k8s 容器内部没法设置防火墙，只能在系统设置，我的思路是 获取ingress-nginx的日
志(python k8s api或者直接kubectl log获取日志)，然后分析日志，通过ingress登录的
日志长这样：
[154.83.14.89] [13/Dec/2022:12:21:52 +0000] TCP 200 1637 1250 0.316
ip 时间 tcp send recv duration
所以不成功的登录一一般都是send/recv<3000字节的，碰到这个，我就记录下来，然后出
现三次同一ip的，就在防火墙封掉ip。
代码写好了，测试的时候发现问题了，k8s完全掌控了iptables，本来我是
iptables -N fail2ban
iptables -I INPUT 1 -j fail2ban
先让fail2ban规则处理的，但是后来想iptables应该是先经过PREROUTING，直接到docker
镜像内部了，从iptables -nL -t nat也能看到，所以规则变成
iptables -t nat -N fail2ban
iptables -t nat -I PREROUTING -j fail2ban
而nat里，是不能drop和reject的，只能把包导到一个不存在的ip -j DNAT --to 0.0.0.1
代码测试成功了，iptables fail2ban的规则也创建了，结果发现某些ip被ban了之后，仍
然能继续访问2222端口，再用iptables一看，
cali-PREROUTING这个PREROUTING的target又排在第一了，看来k8s后台对iptables做了监
控,cali-PREROUTING 始终会排第一，把对ingress暴露的端口的包先转到docker里处理了
。
这个规则太复杂，我不想在DOCKER或者cali-PREROUTING里直接添加规则。
这时候我想到了k8s的networkpolicy的ingress规则，可以限制对pod端口的访问，直接用
python kubernetes api，写了读日志以及patch networkpolicy ingress yaml的接口，
结果发现问题又来了，networkpolicy的ingress规则，只支持默认全部deny，然后加白名
单，而我需要的是默认全部allow，然后设置黑名单。又去看了istio相关设置，同样只支
持白名单，不支持黑名单，郁闷。
明天继续研究。
--
FROM 120.229.14.*
首先k8s会把cali始终放在PREROUTING的第一，这样所有包优先被转发到docker里
# iptables -t nat -nL PREROUTING
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
cali-PREROUTING all -- 0.0.0.0/0 0.0.0.0/0 /*
cali:6gwbT8clXdHdC1b1 */
KUBE-SERVICES all -- 0.0.0.0/0 0.0.0.0/0 /*
kubernetes service portals */
DOCKER all -- 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match
dst-type LOCAL
CNI-HOSTPORT-DNAT all -- 0.0.0.0/0 0.0.0.0/0
ADDRTYPE match dst-type LOCAL
就算我把fail2ban用
iptables -t nat -I PREROUTING 1 -j fail2ban
过一段时间k8s还会把cali排在第一
https://github.com/projectcalico/calico/issues/2672
这里面关系太复杂，生产环境，我还不敢直接在其他CHAIN里增加规则，不知道会出什么
后果。
【 在 hyoga 的大作中提到: 】
: 不懂，转发一个供参考：
: Docker安装了两个定制的iptables链，名为DOCKER-USER和DOCKER，它确保传入的数据
: 包总是首先由这两个链检查。
: Docker的所有iptables规则都被添加到Docker链中，不要手动操作该链条。
: ...................
--
FROM 119.139.196.*
试了一下，把fail2ban放到INPUT里的KUBE-FIREWALL， OUTPUT 都没啥用，其他的都不敢
改了
【 在 qlogic 的大作中提到: 】
: 首先k8s会把cali始终放在PREROUTING的第一，这样所有包优先被转发到docker里
: # iptables -t nat -nL PREROUTING
: Chain PREROUTING (policy ACCEPT)
: ...................
--
FROM 119.139.196.*
突然想到ingress的白名单怎么变成黑名单了，这样
spec:
ingress:
- from:
- ipBlock:
cidr: 0.0.0.0/0
except:
- 10.42.0.229/32
podSelector: {}
policyTypes:
- Ingress
先允许所有，再开expect，except里就是黑名单了
【 在 qlogic 的大作中提到: 】
: 试了一下，把fail2ban放到INPUT里的KUBE-FIREWALL， OUTPUT 都没啥用，其他的都不
: 敢
: 改了
--
FROM 119.139.196.*
fail2ban的目的就是不想在应用层看到那么多尝试连接的日志
【 在 igentoo 的大作中提到: 】
: ingress自己就有这功能吧，看看kong或者trafik
: 发自「今日水木 on iPhone 1」
--
FROM 120.229.14.*
话说回来了，这互联网环境太恶劣了，
10.42.0.182是ingress网关的ip
可以看到，部署fail2ban后，现在一小时的非法访问量还没有之前一分钟的多。
这是部署之前的日志
2022-12-17_08:17:10.80548 Disconnected from invalid user demo1 10.42.0.182
port 51348 [preauth]
2022-12-17_08:17:24.07441 User root from 10.42.0.182 not allowed because not
listed in AllowUsers
2022-12-17_08:17:24.80963 Received disconnect from 10.42.0.182 port 51494:11:
Bye Bye [preauth]
2022-12-17_08:17:24.80965 Disconnected from invalid user root 10.42.0.182
port 51494 [preauth]
2022-12-17_08:17:27.59827 Invalid user demo from 10.42.0.182 port 51574
2022-12-17_08:17:27.82994 Received disconnect from 10.42.0.182 port 51574:11:
Bye Bye [preauth]
2022-12-17_08:17:27.82996 Disconnected from invalid user demo 10.42.0.182
port 51574 [preauth]
2022-12-17_08:17:36.15531 User root from 10.42.0.182 not allowed because not
listed in AllowUsers
2022-12-17_08:17:36.30548 Received disconnect from 10.42.0.182 port 51684:11:
Bye Bye [preauth]
2022-12-17_08:17:36.30551 Disconnected from invalid user root 10.42.0.182
port 51684 [preauth]
2022-12-17_08:17:42.78346 Invalid user hd from 10.42.0.182 port 51770
2022-12-17_08:17:42.96673 Received disconnect from 10.42.0.182 port 51770:11:
Bye Bye [preauth]
2022-12-17_08:17:42.96675 Disconnected from invalid user hd 10.42.0.182 port
51770 [preauth]
2022-12-17_08:17:44.25586 Invalid user eden from 10.42.0.182 port 51774
2022-12-17_08:17:44.64120 Received disconnect from 10.42.0.182 port 51774:11:
Bye Bye [preauth]
2022-12-17_08:17:44.64123 Disconnected from invalid user eden 10.42.0.182
port 51774 [preauth]
2022-12-17_08:17:53.36127 Invalid user pascal from 10.42.0.182 port 51896
2022-12-17_08:17:53.77202 Received disconnect from 10.42.0.182 port 51896:11:
Bye Bye [preauth]
2022-12-17_08:17:53.77204 Disconnected from invalid user pascal 10.42.0.182
port 51896 [preauth]
2022-12-17_08:17:55.40639 User root from 10.42.0.182 not allowed because not
listed in AllowUsers
2022-12-17_08:17:55.64848 Received disconnect from 10.42.0.182 port 51934:11:
Bye Bye [preauth]
2022-12-17_08:17:55.64851 Disconnected from invalid user root 10.42.0.182
port 51934 [preauth]
2022-12-17_08:18:01.34122 Invalid user lms from 10.42.0.182 port 51994
2022-12-17_08:18:02.19226 Received disconnect from 10.42.0.182 port 51994:11:
Bye Bye [preauth]
2022-12-17_08:18:02.19229 Disconnected from invalid user lms 10.42.0.182 port
51994 [preauth]
2022-12-17_08:18:06.93732 User root from 10.42.0.182 not allowed because not
listed in AllowUsers
2022-12-17_08:18:07.01381 Received disconnect from 10.42.0.182 port 52090:11:
Bye Bye [preauth]
2022-12-17_08:18:07.01383 Disconnected from invalid user root 10.42.0.182
port 52090 [preauth]
2022-12-17_08:18:13.99828 User root from 10.42.0.182 not allowed because not
listed in AllowUsers
2022-12-17_08:18:14.18827 Received disconnect from 10.42.0.182 port 52170:11:
Bye Bye [preauth]
这是部署之后的日志
2022-12-18_13:42:51.07126 Invalid user alpha from 10.42.0.182 port 34500
2022-12-18_13:42:51.85064 Received disconnect from 10.42.0.182 port 34500:11:
Bye Bye [preauth]
2022-12-18_13:42:51.85067 Disconnected from invalid user alpha 10.42.0.182
port 34500 [preauth]
2022-12-18_13:43:10.93266 Invalid user db2inst from 10.42.0.182 port 34760
2022-12-18_13:43:10.97628 Received disconnect from 10.42.0.182 port 34760:11:
Bye Bye [preauth]
2022-12-18_13:43:10.97632 Disconnected from invalid user db2inst 10.42.0.182
port 34760 [preauth]
2022-12-18_13:46:15.33125 Invalid user user from 10.42.0.182 port 37140
2022-12-18_13:46:31.66329 Connection closed by invalid user user 10.42.0.182
port 37140 [preauth]
2022-12-18_13:51:12.80500 Invalid user testuser from 10.42.0.182 port 41006
2022-12-18_13:51:13.13223 Connection closed by invalid user testuser
10.42.0.182 port 41006 [preauth]
2022-12-18_13:51:52.89945 Invalid user sftp from 10.42.0.182 port 41520
2022-12-18_13:51:53.08527 Received disconnect from 10.42.0.182 port 41520:11:
Bye Bye [preauth]
2022-12-18_13:51:53.08528 Disconnected from invalid user sftp 10.42.0.182
port 41520 [preauth]
2022-12-18_13:53:53.53812 Invalid user testuser from 10.42.0.182 port 43082
2022-12-18_13:53:53.83396 Connection closed by invalid user testuser
10.42.0.182 port 43082 [preauth]
2022-12-18_14:05:14.61314 Invalid user ahmed from 10.42.0.182 port 51884
2022-12-18_14:05:17.93528 Disconnecting invalid user ahmed 10.42.0.182 port
51884: Change of username or service not allowed: (ahmed,ssh-connection) ->
(ai,ssh-connection) [preauth]
2022-12-18_14:05:24.37588 Invalid user ai from 10.42.0.182 port 51968
2022-12-18_14:05:27.69318 Disconnecting invalid user ai 10.42.0.182 port
51968: Change of username or service not allowed: (ai,ssh-connection) ->
(aiden,ssh-connection) [preauth]
2022-12-18_14:05:36.28860 Invalid user aiden from 10.42.0.182 port 52104
2022-12-18_14:05:39.61187 Disconnecting invalid user aiden 10.42.0.182 port
52104: Change of username or service not allowed: (aiden,ssh-connection) ->
(aim,ssh-connection) [preauth]
2022-12-18_14:05:45.89705 Invalid user aim from 10.42.0.182 port 52252
2022-12-18_14:05:49.21225 Disconnecting invalid user aim 10.42.0.182 port
52252: Change of username or service not allowed: (aim,ssh-connection) ->
(ai,ssh-connection) [preauth]
2022-12-18_14:05:56.23645 Invalid user ai from 10.42.0.182 port 52384
2022-12-18_14:05:59.54457 Disconnecting invalid user ai 10.42.0.182 port
52384: Change of username or service not allowed: (ai,ssh-connection) ->
(airflow,ssh-connection) [preauth]
2022-12-18_14:06:06.61685 Invalid user airflow from 10.42.0.182 port 52508
2022-12-18_14:06:09.92393 Connection closed by invalid user airflow
10.42.0.182 port 52508 [preauth]
2022-12-18_14:31:05.77675 User root from 10.42.0.182 not allowed because not
listed in AllowUsers
2022-12-18_14:31:05.99445 Received disconnect from 10.42.0.182 port 43498:11:
Bye Bye [preauth]
2022-12-18_14:31:05.99448 Disconnected from invalid user root 10.42.0.182
port 43498 [preauth]
可以看到，现在一小时的非法访问量还没有之前一分钟的多。
【 在 qlogic 的大作中提到: 】
: fail2ban的目的就是不想在应用层看到那么多尝试连接的日志
--
修改:qlogic FROM 119.139.196.*
FROM 119.139.196.*