☰

Tomcat servlet 无法完成 websocket 连接

tag-icon • tag-icon apache-2.4 websocket tomcat9
Tomcat servlet 无法完成 websocket 连接

我正在将一组正在工作的 servlet 从一台服务器移动到另一台服务器

Old server, Centos6, Apache 2.2, Tomcat 9
New Server, Centos7, Apache 2.4, Tomcat 9

我在旧服务器上运行了大约 5 个 servlet，除了一个有 3 个 websocket 连接的 servlet 外，其他所有东西都运行正常。除了 websocket 之外，其他一切都在 servlet 中运行。

Tomcat 在 apache 后面进行反向代理。

我对其中一个反向代理的配置

# websocket proxy
<Location "/admin/AdminConsole">
 ProxyPass  wss://localhost:8080/admin/AdminConsole
 ProxyPassReverse wss://localhost:8080/admin/AdminConsole
 Require all granted
</Location>

来自 apache error_log 的输出 LogLevel debug proxy:trace5

[Wed Jul 01 23:10:45.963246 2020] [proxy:trace2] [pid 25640] proxy_util.c(1985): [client 174.30.215.226:46978] http: found worker http://localhost:8080/admin/AdminConsole for http://localhost:8080/admin/AdminConsole?userName=derricks&source=web
[Wed Jul 01 23:10:45.963335 2020] [proxy:debug] [pid 25640] mod_proxy.c(1123): [client 174.30.215.226:46978] AH01143: Running scheme http handler (attempt 0)
[Wed Jul 01 23:10:45.963372 2020] [proxy:debug] [pid 25640] proxy_util.c(2203): AH00942: HTTP: has acquired connection for (localhost)
[Wed Jul 01 23:10:45.963388 2020] [proxy:debug] [pid 25640] proxy_util.c(2256): [client 174.30.215.226:46978] AH00944: connecting http://localhost:8080/admin/AdminConsole?userName=derricks&source=web to localhost:8080
[Wed Jul 01 23:10:45.966024 2020] [proxy:debug] [pid 25640] proxy_util.c(2426): [client 174.30.215.226:46978] AH00947: connected /admin/AdminConsole?userName=derricks&source=web to localhost:8080
[Wed Jul 01 23:10:45.966077 2020] [proxy:trace2] [pid 25640] proxy_util.c(2768): HTTP: fam 10 socket created to connect to localhost
[Wed Jul 01 23:10:45.966734 2020] [proxy:debug] [pid 25640] proxy_util.c(2802): AH02824: HTTP: connection established with [::1]:8080 (localhost)
[Wed Jul 01 23:10:45.966781 2020] [proxy:debug] [pid 25640] proxy_util.c(2942): AH00962: HTTP: connection complete to [::1]:8080 (localhost)
[Wed Jul 01 23:10:45.972425 2020] [proxy:debug] [pid 25640] proxy_util.c(2218): AH00943: http: has released connection for (localhost)

最后一行显示“http”已释放连接？那是 Apache 吗？

[Wed Jul 01 23:10:45.972425 2020] [proxy:debug] [pid 25640] proxy_util.c(2218): AH00943: http: has released connection for (localhost)

apache access_log 输出 404

174.30.215.226 - - [01/Jul/2020:23:10:45 -0400] "GET /admin/AdminConsole?userName=derricks&source=web HTTP/1.1" 404 473 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:77.0) Gecko/20100101 Firefox/77.0" 1611 1025

看起来 Apache 可能正在断开连接？但是为什么呢？

我删除了直接通过 URL:8080 连接到 tomcat 的反向代理，并且 websockets 正常。

看来 Apache 反向代理从 2.2 到 2.4 出了问题。

编辑 # 1

以下是运行 2.2 版服务器和我的新 2.4 版服务器在反向代理期间的日志（LogLevel debug）

This is the new server
[Thu Jul 02 13:39:27.812027 2020] [proxy_fcgi:debug] [pid 53009] mod_proxy_fcgi.c(972): [client 174.30.215.226:38854] AH01076: url: http://localhost:8080/admin/AdminConsole?userName=derricks&source=web proxyname: (null) proxyport: 0
[Thu Jul 02 13:39:27.812040 2020] [proxy_fcgi:debug] [pid 53009] mod_proxy_fcgi.c(975): [client 174.30.215.226:38854] AH01077: declining URL http://localhost:8080/admin/AdminConsole?userName=derricks&source=web
[Thu Jul 02 13:39:27.812062 2020] [proxy:debug] [pid 53009] proxy_util.c(2203): AH00942: HTTP: has acquired connection for (localhost)
[Thu Jul 02 13:39:27.812078 2020] [proxy:debug] [pid 53009] proxy_util.c(2256): [client 174.30.215.226:38854] AH00944: connecting http://localhost:8080/admin/AdminConsole?userName=derricks&source=web to localhost:8080
[Thu Jul 02 13:39:27.813819 2020] [proxy:debug] [pid 53009] proxy_util.c(2426): [client 174.30.215.226:38854] AH00947: connected /admin/AdminConsole?userName=derricks&source=web to localhost:8080
[Thu Jul 02 13:39:27.814389 2020] [proxy:debug] [pid 53009] proxy_util.c(2802): AH02824: HTTP: connection established with [::1]:8080 (localhost)
[Thu Jul 02 13:39:27.814476 2020] [proxy:debug] [pid 53009] proxy_util.c(2942): AH00962: HTTP: connection complete to [::1]:8080 (localhost)
[Thu Jul 02 13:39:27.818830 2020] [proxy:debug] [pid 53009] proxy_util.c(2218): AH00943: http: has released connection for (localhost)
[Thu Jul 02 13:39:27.819281 2020] [deflate:debug] [pid 53009] mod_deflate.c(849): [client 174.30.215.226:38854] AH01384: Zlib: Compressed 1096 to 457 : URL /admin/AdminConsole

This is the old working server
[Thu Jul 02 13:33:11 2020] [debug] mod_proxy_wstunnel.c(91): [client 174.30.215.226] canonicalising URL //localhost:8080/admin/AdminConsole
[Thu Jul 02 13:33:11 2020] [debug] proxy_util.c(1508): [client 174.30.215.226] proxy: wss: found worker wss://localhost:8080/admin/AdminConsole for wss://localhost:8080/admin/AdminConsole?userName=derricks&source=web
[Thu Jul 02 13:33:11 2020] [debug] mod_proxy.c(1028): Running scheme wss handler (attempt 0)
[Thu Jul 02 13:33:11 2020] [debug] mod_proxy_http.c(1963): proxy: HTTP: declining URL wss://localhost:8080/admin/AdminConsole?userName=derricks&source=web
[Thu Jul 02 13:33:11 2020] [debug] mod_proxy_wstunnel.c(388): [client 174.30.215.226] AH02451: serving URL wss://localhost:8080/admin/AdminConsole?userName=derricks&source=web
[Thu Jul 02 13:33:11 2020] [debug] proxy_util.c(2013): proxy: WSS: has acquired connection for (localhost)
[Thu Jul 02 13:33:11 2020] [debug] proxy_util.c(2069): proxy: connecting wss://localhost:8080/admin/AdminConsole?userName=derricks&source=web to localhost:8080
[Thu Jul 02 13:33:11 2020] [debug] proxy_util.c(2195): proxy: connected /admin/AdminConsole?userName=derricks&source=web to localhost:8080
[Thu Jul 02 13:33:11 2020] [debug] proxy_util.c(2446): proxy: WSS: fam 2 socket created to connect to localhost
[Thu Jul 02 13:33:11 2020] [debug] proxy_util.c(2578): proxy: WSS: connection complete to 127.0.0.1:8080 (localhost)
[Thu Jul 02 13:33:11 2020] [debug] mod_proxy_wstunnel.c(236): [client 174.30.215.226] sending request
[Thu Jul 02 13:33:11 2020] [debug] mod_proxy_wstunnel.c(254): [client 174.30.215.226] setting up poll()
[Thu Jul 02 13:33:11 2020] [debug] mod_proxy_wstunnel.c(299): [client 174.30.215.226] AH02445: woke from poll(), i=1
[Thu Jul 02 13:33:11 2020] [debug] mod_proxy_wstunnel.c(308): [client 174.30.215.226] AH02446: sock was readable
[Thu Jul 02 13:33:11 2020] [debug] mod_proxy_wstunnel.c(175): [client 174.30.215.226] wstunnel_transfer complete

最明显的区别是 2.2 正常工作的服务器代理到 wss: 地址，而 2.4 损坏的服务器代理到 http: 地址。我不知道这是日志记录错误还是代理服务错误。我根本没看到损坏的服务器中使用了 wstunnel？

编辑#2

最后终于出现了一种刹车。

我注意到在 web 套接字地址上调用了 proxy_fcgi 并且失败了，所以我删除了它的 LoadModule，现在调用了 proxy_wstunnel，尽管它失败了，并出现了 [提示：SSLProxyEngine]

因此我在 reverseProxy.conf 的顶部启用了 SSProxyEngine

SSLProxyEngine 已开启

现在它变得更远了，实际上连接了 websocket，但它在 SSL 模块中失败了：

以下是日志

[Thu Jul 02 17:55:49.180768 2020] [proxy:debug] [pid 61402] mod_proxy.c(1123): [client 174.30.215.226:42470] AH01143: Running scheme wss handler (attempt 0)
[Thu Jul 02 17:55:49.180823 2020] [proxy_http:debug] [pid 61402] mod_proxy_http.c(1930): [client 174.30.215.226:42470] AH01113: HTTP: declining URL wss://localhost:8080/AdminConsole?userName=derricks&source=web
[Thu Jul 02 17:55:49.180842 2020] [proxy_scgi:debug] [pid 61402] mod_proxy_scgi.c(517): [client 174.30.215.226:42470] AH00865: declining URL wss://localhost:8080/AdminConsole?userName=derricks&source=web
[Thu Jul 02 17:55:49.180857 2020] [proxy_wstunnel:debug] [pid 61402] mod_proxy_wstunnel.c(336): [client 174.30.215.226:42470] AH02451: serving URL wss://localhost:8080/AdminConsole?userName=derricks&source=web
[Thu Jul 02 17:55:49.180874 2020] [proxy:debug] [pid 61402] proxy_util.c(2203): AH00942: WSS: has acquired connection for (localhost)
[Thu Jul 02 17:55:49.180889 2020] [proxy:debug] [pid 61402] proxy_util.c(2256): [client 174.30.215.226:42470] AH00944: connecting wss://localhost:8080/AdminConsole?userName=derricks&source=web to localhost:8080
[Thu Jul 02 17:55:49.182986 2020] [proxy:debug] [pid 61402] proxy_util.c(2426): [client 174.30.215.226:42470] AH00947: connected /AdminConsole?userName=derricks&source=web to localhost:8080
[Thu Jul 02 17:55:49.183216 2020] [proxy:debug] [pid 61402] proxy_util.c(2802): AH02824: WSS: connection established with [::1]:8080 (localhost)
[Thu Jul 02 17:55:49.183269 2020] [proxy:debug] [pid 61402] proxy_util.c(2942): AH00962: WSS: connection complete to [::1]:8080 (localhost)
[Thu Jul 02 17:55:49.183293 2020] [ssl:info] [pid 61402] [remote ::1:8080] AH01964: Connection to child 0 established (server mbepapers.org:443)
[Thu Jul 02 17:55:49.188928 2020] [ssl:info] [pid 61402] [remote ::1:8080] AH02003: SSL Proxy connect failed
[Thu Jul 02 17:55:49.189045 2020] [ssl:info] [pid 61402] SSL Library Error: error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol
[Thu Jul 02 17:55:49.189063 2020] [ssl:info] [pid 61402] [remote ::1:8080] AH01998: Connection closed to child 0 with abortive shutdown (server mbepapers.org:443)
[Thu Jul 02 17:55:49.189113 2020] [ssl:info] [pid 61402] [remote ::1:8080] AH01997: SSL handshake failed: sending 502
[Thu Jul 02 17:55:49.189154 2020] [proxy_wstunnel:debug] [pid 61402] mod_proxy_wstunnel.c(257): [client 174.30.215.226:42470] AH02445: woke from poll(), i=1
[Thu Jul 02 17:55:49.189167 2020] [proxy_wstunnel:debug] [pid 61402] mod_proxy_wstunnel.c(266): [client 174.30.215.226:42470] AH02446: sock was readable
[Thu Jul 02 17:55:49.189180 2020] [proxy_wstunnel:debug] [pid 61402] mod_proxy_wstunnel.c(131): (103)Software caused connection abort: [client 174.30.215.226:42470] AH02442: error on sock - ap_get_brigade
[Thu Jul 02 17:55:49.189234 2020] [proxy:debug] [pid 61402] proxy_util.c(2218): AH00943: WSS: has released connection for (localhost)

快完成了！现在 proxy.c 正在运行 wss 方案，URL 地址是

wss://localhost:8080/AdminConsole?用户名=derricks&source=web

这是我的客户一直发送的 URL！

错误出在 SSL 模块中

[Thu Jul 02 23:53:57.947613 2020] [ssl:info] [pid 7910] [remote ::1:8080] AH02003: SSL Proxy connect failed
[Thu Jul 02 23:53:57.947758 2020] [ssl:info] [pid 7910] SSL Library Error: error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol
[Thu Jul 02 23:53:57.947778 2020] [ssl:info] [pid 7910] [remote ::1:8080] AH01998: Connection closed to child 0 with abortive shutdown (server mbepapers.org:443)
[Thu Jul 02 23:53:57.948036 2020] [ssl:info] [pid 7910] [remote ::1:8080] AH01997: SSL handshake failed: sending 502

SSL 库错误：错误：140770FC：SSL 例程：SSL23_GET_SERVER_HELLO：未知协议

正在寻找解决方案???

编辑3

我可以使用 https:/IP 连接到此服务器，因为要替换的旧服务器使用正确的 DNS IP<-> 域名。我安装了该服务器的证书，因为我很快就会交换 IP。我想知道这是否是错误的原因？

编辑4

仍在寻找解决方案。当我尝试代理 websocket 时，httpd error_log 中出现此错误，日志级别为 debug

[Sat Jul 11 14:27:23.978213 2020] [proxy:debug] [pid 9147] mod_proxy.c(1123): [client 174.30.215.226:54434] AH01143: Running scheme wss handler (attempt 0)
[Sat Jul 11 14:27:23.978266 2020] [proxy_wstunnel:debug] [pid 9147] mod_proxy_wstunnel.c(336): [client 174.30.215.226:54434] AH02451: serving URL wss://localhost:8080/admin/AdminConsole?userName=derricks&source=app
[Sat Jul 11 14:27:23.978302 2020] [proxy:debug] [pid 9147] proxy_util.c(2203): AH00942: WSS: has acquired connection for (localhost)
[Sat Jul 11 14:27:23.978323 2020] [proxy:debug] [pid 9147] proxy_util.c(2256): [client 174.30.215.226:54434] AH00944: connecting wss://localhost:8080/admin/AdminConsole?userName=derricks&source=app to localhost:8080
[Sat Jul 11 14:27:23.980603 2020] [proxy:debug] [pid 9147] proxy_util.c(2426): [client 174.30.215.226:54434] AH00947: connected /admin/AdminConsole?userName=derricks&source=app to localhost:8080
[Sat Jul 11 14:27:23.981093 2020] [proxy:debug] [pid 9147] proxy_util.c(2802): AH02824: WSS: connection established with [::1]:8080 (localhost)
[Sat Jul 11 14:27:23.981205 2020] [proxy:debug] [pid 9147] proxy_util.c(2942): AH00962: WSS: connection complete to [::1]:8080 (localhost)
[Sat Jul 11 14:27:23.981355 2020] [ssl:info] [pid 9147] [remote ::1:8080] AH01964: Connection to child 0 established (server www.mbepapers.org:443)
[Sat Jul 11 14:27:23.983594 2020] [ssl:info] [pid 9147] [remote ::1:8080] AH02003: SSL Proxy connect failed
[Sat Jul 11 14:27:23.983814 2020] [ssl:info] [pid 9147] SSL Library Error: error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol
[Sat Jul 11 14:27:23.983871 2020] [ssl:info] [pid 9147] [remote ::1:8080] AH01998: Connection closed to child 0 with abortive shutdown (server www.mbepapers.org:443)
[Sat Jul 11 14:27:23.983907 2020] [ssl:info] [pid 9147] [remote ::1:8080] AH01997: SSL handshake failed: sending 502
[Sat Jul 11 14:27:23.983969 2020] [proxy_wstunnel:debug] [pid 9147] mod_proxy_wstunnel.c(257): [client 174.30.215.226:54434] AH02445: woke from poll(), i=1
[Sat Jul 11 14:27:23.984005 2020] [proxy_wstunnel:debug] [pid 9147] mod_proxy_wstunnel.c(266): [client 174.30.215.226:54434] AH02446: sock was readable
[Sat Jul 11 14:27:23.984019 2020] [proxy_wstunnel:debug] [pid 9147] mod_proxy_wstunnel.c(131): (103)Software caused connection abort: [client 174.30.215.226:54434] AH02442: error on sock - ap_get_brigade
[Sat Jul 11 14:27:23.984092 2020] [proxy:debug] [pid 9147] proxy_util.c(2218): AH00943: WSS: has released connection for (localhost)

相关台词

[Sat Jul 11 14:27:23.983594 2020] [ssl:info] [pid 9147] [remote ::1:8080] AH02003: SSL Proxy connect failed
[Sat Jul 11 14:27:23.983814 2020] [ssl:info] [pid 9147] SSL Library Error: error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol
[Sat Jul 11 14:27:23.983871 2020] [ssl:info] [pid 9147] [remote ::1:8080] AH01998: Connection closed to child 0 with abortive shutdown (server www.mbepapers.org:443)
[Sat Jul 11 14:27:23.983907 2020] [ssl:info] [pid 9147] [remote ::1:8080] AH01997: SSL handshake failed: sending 502

有人建议我将反向代理移到 vhost 部分内，因此这里

<VirtualHost xx.xx.xx.xx:443>
 RewriteEngine on
 ServerName www.mbepapers.org
 ServerAlias www.mbepapers.org
 SSLEngine on
 SSLProxyEngine on
 SSLCertificateFile /var/www/httpd-cert/mbepapers.org.crt
 SSLCertificateKeyFile /var/www/httpd-cert/mbepapers.org.key
 SSLCertificateChainFile /var/www/httpd-cert/intermediate.crt
 SSLCaCertificateFile /var/www/httpd-cert/intermediate.crt

 ProxyRequests off

 <Location "/admin/AdminConsole">
   ProxyPass  wss://localhost:8080/admin/AdminConsole
   ProxyPassReverse  wss://localhost:8080/admin/AdminConsole
   Require all granted
 </Location>

答案1

修复方法很简单。我更改了反向代理，使用不安全的 websocket 连接到我的 tomcat 服务器

这导致 SSL 代理身份验证失败

# websocket proxy
<Location "/admin/AdminConsole">
  ProxyPass  wss://localhost:8080/admin/AdminConsole
  ProxyPassReverse  wss://localhost:8080/admin/AdminConsole
  Require all granted
</Location>

这可以避免 SSL 代理身份验证的需要

# websocket proxy
<Location "/admin/AdminConsole">
  ProxyPass  ws://localhost:8080/admin/AdminConsole
  ProxyPassReverse  ws://localhost:8080/admin/AdminConsole
  Require all granted
</Location>

从客户端，我使用的是安全的 websocket 地址，而 apache 现在将其代理到不安全的 websocket。这没问题，因为我在互联网上是安全的，而只有在本地主机上是不安全的。

这是客户端上的连接地址

WSuri = new URI("wss://"+host+"/admin/AdminConsole?userName="+userName+"&source=app");

Apache 2.2 没有问题，而 2.4.6 却有问题，这仍然很奇怪。不过我读过几篇帖子，指出 2.4.6 的安全代理存在问题。

相关内容