使用 ELB 的高可用性 Squid 代理

使用 ELB 的高可用性 Squid 代理

我目前正在尝试设置一个由 ASG、ELB 和 EC2 实例组成的 HA squid 代理。我已将 Internet Explorer 上的 LAN 设置中的代理服务器设置设置为 ELB DNS 名称。当尝试从允许的 URL 列表中加载网页时,我收到以下错误消息,其中实例配置为使用代理:

ERROR

The requested URL could not be retrieved

The following error was encountered while trying to retrieve the URL: /

Invalid URL

Some aspect of the requested URL is incorrect.

Some possible problems are:
•Missing or incorrect access protocol (should be “http://” or similar)
•Missing hostname
•Illegal double-escape in the URL-Path
•Illegal character in hostname; underscores are not allowed.
Your cache administrator is root.

问题似乎出在负载均衡器上,只要您将 Internet Explorer 代理设置设为代理私有 DNS 或私有 IP 地址,就不会出现问题,代理可以正常工作,即网页可以正常加载。

这是 squid.conf

            # This file generated from a Chef template.
            # squid/templates/default.squid.conf.erb
            acl manager proto cache_object
            acl localhost src 127.0.0.1/32
            acl to_localhost dst 127.0.0.0/8 0.0.0.0/32
            acl localnet src 10.0.0.0/8     # RFC1918 possible internal network
            acl localnet src 172.16.0.0/12  # RFC1918 possible internal network
            acl localnet src 192.168.0.0/16 # RFC1918 possible internal network
            acl localnet src fc00::/7       # RFC4193 local private network range
            acl localnet src fe80::/10      # RFC4291 link-local (directly-plugged) machine
            acl SSL_ports port 443      # https
            acl SSL_ports port 563      # snews
            acl SSL_ports port 873      # rsync
            acl Safe_ports port 80      # http
            acl Safe_ports port 21      # ftp
            acl Safe_ports port 443     # https
            acl Safe_ports port 70      # gopher
            acl Safe_ports port 210     # wais
            acl Safe_ports port 1025-65535  # unregistered ports
            acl Safe_ports port 280     # http-mgmt
            acl Safe_ports port 488     # gss-http
            acl Safe_ports port 591     # filemaker
            acl Safe_ports port 777     # multiling http
            acl Safe_ports port 631     # cups
            acl Safe_ports port 873     # rsync
            acl Safe_ports port 901     # SWAT
            acl purge method PURGE
            acl CONNECT method CONNECT
            http_access allow all
            # Managed with Chef
            acl web-hosts src all
            acl web-bd dstdomain .amazonaws.com
            acl web-bd dstdomain .chef.io
            acl web-bd dstdomain .rubygems.org
            acl web-bd dstdomain .splunk.com
            acl web-bd dstdomain .bintray.com
            acl web-bd dstdomain .trendmicro.com
            acl web-bd dstdomain .slproweb.com
            acl web-bd dstdomain .fastly.net
            http_access allow web-bd
            # The line below blocks all websites which are not allowed in the squid_urls data bag
            http_access deny !web-bd
            http_access allow manager localhost
            http_access deny manager
            http_access allow purge localhost
            http_access deny purge
            http_access deny !Safe_ports
            http_access deny CONNECT !SSL_ports
            http_access allow localhost
            http_access allow localnet
            http_access deny all
            icp_access allow localnet
            icp_access deny all
            http_port 3130 protocol=HTTP
            hierarchy_stoplist cgi-bin ?
            access_log /var/log/squid/access.log squid
            refresh_pattern     ^ftp:               1440    20%     10080
            refresh_pattern     ^gopher:            1440    0%      1440
            refresh_pattern     -i (/cgi-bin/|\?)       0       0%      0
            refresh_pattern     (Release|Package(.gz)*)$    0       20%     2880
            # refresh_pattern           \.deb$              1440    20%     10080
            # refresh_pattern           \.rpm$              1440    20%     10080
            # refresh_pattern           \.iso$              1440    20%     10080
            # refresh_pattern           \.$         1440    20%     10080
            # refresh_pattern           .               0       20%     4320
            hosts_file /etc/hosts
            maximum_object_size 1024 MB
            coredump_dir /var/spool/squid
            cache_mem 0 MB
            debug_options ALL

我打开了 ELB 日志记录,以下是一些流量:

            2016-09-02T10:58:59.552990Z ha-proxy 10.166.107.198:56190 10.166.106.20:3130 0.000036 0.000908 0.000026 400 400 0 3154 "GET http://youtube.com:3130/ HTTP/1.1" "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:38.0) Gecko/20100101 Firefox/38.0" - -
            2016-09-02T10:58:59.572031Z ha-proxy 10.166.107.198:56190 10.166.106.20:3130 0.000023 0.000689 0.000018 400 400 0 3182 "GET http://www.squid-cache.org:3130/Artwork/SN.png HTTP/1.1" "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:38.0) Gecko/20100101 Firefox/38.0" - -
            2016-09-02T10:59:01.844569Z ha-proxy 10.166.107.198:56190 10.166.106.20:3130 0.000046 0.001085 0.000021 400 400 0 3154 "GET http://google.com:3130/ HTTP/1.1" "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:38.0) Gecko/20100101 Firefox/38.0" - -
            2016-09-02T10:59:01.862584Z ha-proxy 10.166.107.198:56190 10.166.106.20:3130 0.00004 0.000839 0.000023 400 400 0 3182 "GET http://www.squid-cache.org:3130/Artwork/SN.png HTTP/1.1" "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:38.0) Gecko/20100101 Firefox/38.0" - -

从 squid 访问日志中:

            1472813779.899      0 10.166.106.80 NONE/400 4075 NONE error:invalid-request - NONE/- text/html
            1472813793.898      0 10.166.106.80 NONE/400 4075 NONE error:invalid-request - NONE/- text/html
            1472813821.915      0 10.166.106.80 NONE/400 4075 NONE error:invalid-request - NONE/- text/html
            1472813828.898      0 10.166.106.80 NONE/400 4075 NONE error:invalid-request - NONE/- text/html
            1472813835.898      0 10.166.106.80 NONE/400 4075 NONE error:invalid-request - NONE/- text/html
            1472813849.898      0 10.166.106.80 NONE/400 4075 NONE error:invalid-request - NONE/- text/html
            1472813856.898      0 10.166.106.80 NONE/400 4075 NONE error:invalid-request - NONE/- text/html
            1472813863.899      0 10.166.106.80 NONE/400 4075 NONE error:invalid-request - NONE/- text/html
            1472813870.900      0 10.166.106.80 NONE/400 4075 NONE error:invalid-request - NONE/- text/html
            1472813884.876      0 10.166.106.29 NONE/400 4076 NONE error:invalid-request - NONE/- text/html
            1472813898.899      0 10.166.106.80 NONE/400 4075 NONE error:invalid-request - NONE/- text/html
            1472813939.552      0 10.166.106.29 NONE/400 3533 GET / - NONE/- text/html
            1472813939.570      0 10.166.106.29 NONE/400 3561 GET /Artwork/SN.png - NONE/- text/html
            1472813940.899      0 10.166.106.80 NONE/400 4075 NONE error:invalid-request - NONE/- text/html
            1472813941.843      0 10.166.106.29 NONE/400 3533 GET / - NONE/- text/html
            1472813941.861      0 10.166.106.29 NONE/400 3561 GET /Artwork/SN.png - NONE/- text/html
            1472813961.899      0 10.166.106.80 NONE/400 4075 NONE error:invalid-request - NONE/- text/html
            1472817013.903      0 10.166.106.80 NONE/400 4075 NONE error:invalid-request - NONE/- text/html
            1472817020.898      0 10.166.106.80 NONE/400 4075 NONE error:invalid-request - NONE/- text/html
            1472817026.226      0 10.166.106.80 NONE/400 4076 NONE error:invalid-request - NONE/- text/html
            1472817026.332      0 10.166.106.80 NONE/400 4076 NONE error:invalid-request - NONE/- text/html
            1472817034.899      0 10.166.106.80 NONE/400 4075 NONE error:invalid-request - NONE/- text/html
            1472817048.898      0 10.166.106.80 NONE/400 4075 NONE error:invalid-request - NONE/- text/html
            1472817083.899      0 10.166.106.80 NONE/400 4075 NONE error:invalid-request - NONE/- text/html
            1472817097.898      0 10.166.106.80 NONE/400 4075 NONE error:invalid-request - NONE/- text/html
            1472817104.898      0 10.166.106.80 NONE/400 4075 NONE error:invalid-request - NONE/- text/html
            1472817153.904      0 10.166.106.80 NONE/400 4075 NONE error:invalid-request - NONE/- text/html
            1472817160.899      0 10.166.106.80 NONE/400 4075 NONE error:invalid-request - NONE/- text/html
            1472817174.898      0 10.166.106.80 NONE/400 4075 NONE error:invalid-request - NONE/- text/html
            1472817195.920      0 10.166.106.29 NONE/400 4076 NONE error:invalid-request - NONE/- text/html
            1472817202.898      0 10.166.106.80 NONE/400 4075 NONE error:invalid-request - NONE/- text/html
            1472817223.898      0 10.166.106.80 NONE/400 4075 NONE error:invalid-request - NONE/- text/html
            1472817237.900      0 10.166.106.80 NONE/400 4075 NONE error:invalid-request - NONE/- text/html
            1472817251.899      0 10.166.106.80 NONE/400 4075 NONE error:invalid-request - NONE/- text/html
            1472817265.899      0 10.166.106.80 NONE/400 4075 NONE error:invalid-request - NONE/- text/html
            1472817275.160      0 10.166.106.80 NONE/400 3533 GET / - NONE/- text/html
            1472817275.193      0 10.166.106.80 NONE/400 3561 GET /Artwork/SN.png - NONE/- text/html
            1472817278.626      0 10.166.106.29 NONE/400 3677 GET /fwlink/?LinkID=403856&language=en-US&scale=100&contrast=gray - NONE/- text/html
            1472817330.367      0 10.166.106.80 NONE/400 3533 GET / - NONE/- text/html
            1472817330.397      0 10.166.106.80 NONE/400 3561 GET /Artwork/SN.png - NONE/- text/html
            1472817330.448      0 10.166.106.80 NONE/400 3555 GET /favicon.ico - NONE/- text/html
            1472817330.453      0 10.166.106.80 NONE/400 3555 GET /favicon.ico - NONE/- text/html
            1472817337.300      0 10.166.106.29 NONE/400 3533 GET / - NONE/- text/html
            1472817337.334      0 10.166.106.29 NONE/400 3561 GET /Artwork/SN.png - NONE/- text/html
            1472818532.464      0 10.166.106.29 NONE/400 3533 GET / - NONE/- text/html
            1472818532.478      0 10.166.106.29 NONE/400 3561 GET /Artwork/SN.png - NONE/- text/html
            1472818533.259      0 10.166.106.29 NONE/400 3533 GET / - NONE/- text/html
            1472818533.278      0 10.166.106.29 NONE/400 3561 GET /Artwork/SN.png - NONE/- text/html
            1472818534.108      0 10.166.106.29 NONE/400 3533 GET / - NONE/- text/html
            1472818534.137      0 10.166.106.29 NONE/400 3561 GET /Artwork/SN.png - NONE/- text/html

有人有什么想法吗?我快抓狂了。

答案1

你是对的 - 这是一个与 ELB 相关的问题。

tl;dr - 您可以通过从 HTTP 切换到 TCP 来与后端 squid 服务器通信来解决此问题。

当客户端向代理发送 HTTP 请求时,请求包含绝对 URI(例如 GET http ://host-url HTTP/1.1)。
但是,AWS ELB 强制执行最常见的 HTTP 请求形式并重写请求(GET / HTTP/1.1 HOST www.host-url.com)。
这次重写使得 Squid 无法读取它因此,目前无法使用 ELB HTTP 侦听器和“X-Forwarded-For”HTTP 标头来检索客户端 IP 地址。

为了解决这个问题,我改用了 TCP 协议,因为使用此协议,ELB 会按原样传递请求,而不会插入请求标头。
这样做的一个主要问题是,您无法获得常规 ELB 标头(例如 x-forwarded-for),并且无法记录客户端的 IP 地址(只能记录 ELB 的 IP 地址)。

这个问题有多种解决方案:
1. 使用代理传递协议和 TCP 协议配置 ELB。
你可以阅读更多相关信息这里.
2. 使用 squid 3.5 及以上版本。此版本支持代理传递协议,应从 ELB 解析 URL。阅读更多相关信息这里

相关内容