Kubernetes的Nginx Ingress 0.20之前的版本，upstream的keep-alive不生效

Kubernetes的Nginx Ingress 0.20之前的版本，upstream的keep-alive不生效

1说明

Kubernetes使用​​nginx-ingress-controller​​代理到集群内服务的请求，nginx所在的机器上上有大量的time-wait连接。

抓包发现nginx发起的到upstream连接中只有一个请求，keep-alive​​强调需要设置close​​，所以主要关心特殊情况下的行为）

3.不配置close”请求

先测试一下在不配置1.1和不清除Connection请求头的情况下，请求端在请求头里带上“Connection: close”会怎样。

使用下面的配置文件：

$ cat /etc/nginx/conf.d/echo.com.confupstream echo_upstream{ server 172.16.128.126:8080; keepalive 10;} server { listen 7000 ; listen [::]:7000 ; server_name echo.com; # 在本地host配置域名 keepalive_requests 2000; keepalive_timeout 60s; location / { proxy_pass proxy_1.1; # 故意注释这两行配置，观察下行为# proxy_set_header Connection ""; # }}

使用下面的压测命令：

./wrk -d 2m -c 20 -H "host: echo.com" -H "Connection: close"

在upstream端抓包，发现除了少部分报文抓取不连续的连接，其它所有连接中都只包含一次-d 2m -c 20 -H "host: echo.com"

压测端发出的请求是这样的：

GET / HTTP/1.1host: echo.comHost: 10.10.64.58:7000 HTTP/1.1 200 OKServer: nginx/1.12.2Date: Wed, 08 May 2019 09:01:26 GMTContent-Type: text/plainContent-Length: 379Connection: keep-alive...

而upstream端收到请求是带有close的，一个连接中只有一个请求：

GET / HTTP/1.0Host: echo_upstreamConnection: close HTTP/1.1 200 OKDate: Wed, 08 May 2019 09:08:08 GMTContent-Type: text/plainContent-Length: 379Connection: closeServer: echoserver

5.不配置keep-alive”请求

即使请求端设置了keep-alive，nginx转发给upstream的依然是Connection: close：

GET / HTTP/1.0Host: echo_upstreamConnection: close HTTP/1.1 200 OKDate: Wed, 08 May 2019 11:37:58 GMTContent-Type: text/plainContent-Length: 379Connection: closeServer: echoserver

6.小结

由此得出结论，如果不进行下面设置，无论请求端如何调整，nginx转发给upstream时都不会使用keep-alive：

server { ... location /{ proxy_pass proxy_1.1; proxy_set_header Connection ""; ... }}

7.检查问题环境

先看一下问题环境upstream收到的报文，是HTTP/1.1，带有Connection: close，这个组合有点奇怪，是前面试验中没遇到的：

GET /js/manifest.988fecf548c158ad4ab7.js HTTP/1.1Host: ********Connection: closeX-Real-IP: ********....

检查配置文件发现了问题：

...# Retain the default nginx handling of requests without a "Connection" headermap $$connection_upgrade { default upgrade; '' close;}... # Allow websocket connectionsproxy_set_header Upgrade $ Connection $connection_upgrade;...

在配置模板文件/etc/nginx/template/nginx.tmpl中找到了这段配置的说明，这是nginx 1.3开始提供的​​WebSocket proxying​​代理功能：

{{/* Whenever nginx proxies a request without a "Connection" header, the "Connection" header is set to "close" */}}{{/* when making the target request. This means that you cannot simply use */}}{{/* "proxy_set_header Connection $for WebSocket support because in this case, the */}}{{/* "Connection" header would be set to "" whenever the original request did not have a "Connection" header, */}}{{/* which would mean no "Connection" header would be in the target request. Since this would deviate from */}}{{/* normal nginx behavior we have to use this approach. */}}# Retain the default nginx handling of requests without a "Connection" headermap $$connection_upgrade { default upgrade; '' close;}

上面配置的影响是：当close”（nginx的默认行为）。keepalive因此而无效，之所以是HTTP/1.1，是因为配置文件有这样一样配置：

proxy_ 1.1;

8.修改并验证

在测试环境更新nginx配置，将默认行为设置为“Connection: ““”，看一下能否解决问题。更新后的配置如下：

upstream echo_upstream{ server 172.16.128.126:8080; keepalive 1;} map $$connection_upgrade { default upgrade; '' "";} server { listen 7000 ; listen [::]:7000 ; server_name echo.com; # 在本地host配置域名 keepalive_requests 2000; keepalive_timeout 60s; location / { proxy_pass proxy_set_header Upgrade $ proxy_set_header Connection $connection_upgrade; proxy_1.1;# proxy_set_header Connection ""; }}

再次测试，请求端不使用keep-alive：

./wrk -d 2m -c 20 -H "host: echo.com" -H "Connection: close"

在upstream端抓包，即使请求端不使用keep-alive，nginx转发upstream的时候还是会使用keep-alive:

GET / HTTP/1.1Host: echo_upstream HTTP/1.1 200 OKDate: Wed, 08 May 2019 11:54:00 GMTContent-Type: text/plainTransfer-Encoding: chunkedConnection: keep-aliveServer: echoserver... GET / HTTP/1.1Host: echo_upstream HTTP/1.1 200 OKDate: Wed, 08 May 2019 11:54:00 GMTContent-Type: text/plainTransfer-Encoding: chunkedConnection: keep-aliveServer: echoserver

把keepalive_timeout 60s;调大，会看到请求端停止请求之后，nginx与upstream还有连接。

9.最终结论

刚开始怀疑是因历史原因修改了nginx.tmpl导致的，但是通过比对nginx-ingress-controller 0.24.0、问题环境中0.9.0以及原始的0.9.0中的nginx.tmpl，发现这是0.9.0版本中的一个bug。

0.24.0的配置模板中设置map的时候，会根据$cfg.UpstreamKeepaliveConnections的值做不同设置：

# See $$connection_upgrade { default upgrade; {{ if (gt $cfg.UpstreamKeepaliveConnections 0) }} # See '' ''; {{ else }} '' close; {{ end }}}

0.9.0中则没有考虑keepalive的因素：

map $$connection_upgrade { default upgrade; '' close;}

翻阅​​ChangeLog​​​找到了这个问题的修复记录，是在0.20.0版本修复的：​​make upstream keepalive work for #3098​​。

10.参考

​​Nginx keep-alive​​​​NGINX Ingress Controller​​​​WebSocket proxying​​​​https://lijiaocn.com/%E9%97%AE%E9%A2%98/2019/05/08/nginx-ingress-keep-alive-not-work.html#目录​​

作者：小家电维修

转世燕还故榻,为你衔来二月的花。

版权声明：本文内容由网络用户投稿，版权归原作者所有，本站不拥有其著作权，亦不承担相应法律责任。如果您发现本站中有涉嫌抄袭或描述失实的内容，请联系我们jiasou666@gmail.com 处理，核实后本网站将在24小时内删除侵权内容。