Squid Proxy StoreID 与 pfSense 一起使用

Question

我想知道是否有人可以帮助我了解有关 Squid 的 StoreID 的更多信息？

我已经研究这个问题一段时间了。我让网络缓存加速器工作，但有时我感觉它本身缺乏安全性，或者它可以缓存更多。

它确实有效。

网络缓存/加速器中的点击照片

这是我的配置，我在某些设备上使用 SSL 证书，而有些设备没有证书并以透明模式工作：

acl localhost src 192.168.1.1/32
#cachemgr_passwd disable offline_toggle reconfigure shutdown
#cachemgr_passwd REDACTED all
acl no_miss url_regex -i gateway\.facebook\.com\/ws\/realtime\?
acl no_miss url_regex -i web-chat-e2ee\.facebook\.com\/ws\/chat 
acl CONNECT method CONNECT
acl wuCONNECT dstdomain www.update.microsoft.com
acl wuCONNECT dstdomain sls.microsoft.com
http_access allow CONNECT wuCONNECT localnet
http_access allow CONNECT wuCONNECT localhost
http_access allow windowsupdate localnet
http_access allow windowsupdate localhost
http_access deny manager

acl BrokenButTrustedServers dstdomain "/usr/local/pkg/dstdom.broken"
acl DomainMismatch ssl_error SQUID_X509_V_ERR_DOMAIN_MISMATCH
sslproxy_cert_error allow BrokenButTrustedServers DomainMismatch
sslproxy_cert_error deny all

acl splice_only src 192.168.1.8 #Tasha iPhone
acl splice_only src 192.168.1.10 #Jon iPhone
acl splice_only src 192.168.1.11 #Amazon Fire
acl splice_only src 192.168.1.15 #Tasha HP
acl splice_only src 192.168.1.16 #iPad

acl NoSSLIntercept ssl::server_name_regex -i "/usr/local/pkg/reg.url.nobump"
acl NoBumpDNS dstdomain "/usr/local/pkg/dns.nobump"

acl markBumped annotate_client bumped=true
acl bump_only src 192.168.1.3 #webtv
acl bump_only src 192.168.1.4 #toshiba
acl bump_only src 192.168.1.5 #imac
acl bump_only src 192.168.1.9 #macbook
acl bump_only src 192.168.1.13 #dell

ssl_bump peek step1
miss_access deny no_miss 
ssl_bump splice https_login
ssl_bump splice splice_only
ssl_bump splice NoBumpDNS
ssl_bump splice NoSSLIntercept
ssl_bump bump bump_only markBumped
ssl_bump stare all

acl markedBumped note bumped true
url_rewrite_access deny markedBumped

read_ahead_gap 64 KB
negative_ttl 1 second
connect_timeout 30 seconds
request_timeout 60 seconds
half_closed_clients off
shutdown_lifetime 10 seconds
negative_dns_ttl 1 seconds
ignore_unknown_nameservers on
pipeline_prefetch 100

#acl SSLIntercept ssl::server_name_regex -i "/usr/local/pkg/url.bump"
#ssl_bump bump SSLIntercept

这是我个人的不碰撞文件的副本，这些 URL 只是传递而没有使用缓存，而且只是拼接而没有碰撞。

.dssott.com
.prod-ripcut-delivery.disney-plus.net
.disney.api.edge.bamgrid.com
.disney.playback.edge.bamgrid.com
.disney.my.sentry.io
.hulustream.com
.hulu.com
.hulu.hb.omtrdc.net
.hulu.playback.edge.bamgrid.com
.assetshuluimcom-a.akamaihd.net
.hulu.sc.omtrdc.net
.beacons.extremereach.io
.tubi.video
.tubi.io
.tubitv.com
.a-fds.youborafds01.com
.license.adrise.tv
.amzpvxrayasset-a.akamaihd.net
.pv-cdn.net
.media-amazon.com
.aiv-delivery.net
.unagi.amazon.com
.atv-ps.amazon.com
.pv-cdn.net
.fls-na.amazon.com
.aiv-cdn.net
.c0a299900000.local
.conviva.com
.cdn.office.net
.bitdefender.net
.azure-devices.net
.substrate.office.com
.update.microsoft.com
.update.microsoft.com.akadns.net
.delivery.mp.microsoft.com
.appldnld.apple.com
.configuration.apple.com
.gdmf.apple.com
.mesu.apple.com
.oscdn.apple.com
.osrecovery.apple.com
.skl.apple.com
.swcdn.apple.com
.swdist.apple.com
.swscan.apple.com
.appldnld.apple.com.edgesuite.net
.entrust.net
.digicert.com
.apple-cloudkit.com
.apple-livephotoskit.com
.gc.apple.com
.icloud-content.com
.cdn-apple.com
.icloud.com
.appattest.apple.com
.itunes.apple.com
.mzstatic.com
.itunes.com
.music.apple.com
.app-site-association.networking.apple.com
.xp.apple.com
.play.google.com
.android.com
.google-analytics.com
.googleusercontent.com
.ggpht.com
.dl.google.com
.dl-ssl.google.com
.android.clients.google.com
.android.clients.google.com
.omahaproxy.appspot.com
.payments.google.com
.googleapis.com
.notifications.google.com
.ogs.google.com
.googleapis.com
.privacyportal-bofa.my.onetrust.com
.bankofamerica.com
.mcafee.com
.kaspersky.com
.kaspersky-labs.com
.ml.com
.zoom.us
.teams.microsoft.com
.edge-chat.facebook.com
.internet.speedpay.com
.amazonvideo.com
.unagi-na.amazon.com
.events.data.microsoft.com
.caauthservice.state.gov
.studentaid.gov
.mohela.com
www.whitehouse.gov
www.rcsdk8.org
.rcsdk8.powerschool.com
www.weaveinc.org
.cdn.nintendo.net

这是我的“请勿碰撞”reg ex 文件的副本

#Sites to be splice
(disney\.(content|connections))\.edge\.bamgrid\.com
web-chat-e2ee\.facebook\.com\/ws\/chat  
gateway\.facebook\.com\/ws\/realtime\?
^((alt[0-9]-mtalk\.)|(mtalk\.)|(mtalk-(staging|dev)\.))google\.com
^((gvt)([0-9]))\.com
^(((clients)[0-9])|accounts)\.google\.(com|us)
^(pki|(crl|ocsp)\.pki)\.google\.com
(outlook\.)(office365|office)\.com
infinity-c[0-9][0-9]\.youboranqs[0-9][0-9]\.com

这是我正在使用的自定义 refresh_patterns。

acl getmethod method GET

tls_outgoing_options cipher=HIGH:MEDIUM:!RC4:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS

tls_outgoing_options options=NO_SSLv3,SINGLE_DH_USE,SINGLE_ECDH_USE

acl windowsupdate dstdomain windowsupdate.microsoft.com
acl windowsupdate dstdomain .update.microsoft.com
acl windowsupdate dstdomain download.windowsupdate.com
acl windowsupdate dstdomain redir.metaservices.microsoft.com
acl windowsupdate dstdomain images.metaservices.microsoft.com
acl windowsupdate dstdomain c.microsoft.com
acl windowsupdate dstdomain www.download.windowsupdate.com
acl windowsupdate dstdomain wustat.windows.com
acl windowsupdate dstdomain crl.microsoft.com
acl windowsupdate dstdomain sls.microsoft.com
acl windowsupdate dstdomain productactivation.one.microsoft.com
acl windowsupdate dstdomain ntservicepack.microsoft.com
acl windowsupdate dstdomain dc1-st.ksn.kaspersky-labs.com
acl windowsupdate dstdomain dc1-file.ksn.kaspersky-labs.com
acl windowsupdate dstdomain dc1.ksn.kaspersky-labs.com

acl rewritedoms dstdomain .facebook.com .akamaihd.net .fbcdn.net .google.com .static.com .apple.com .oracle.com .sun.com .java.com .adobe.com .steamstatic.com .steampowered.com .steamcontent.com .google.com

store_id_program /usr/local/libexec/squid/storeid_file_rewrite /var/squid/storeid/storeid_rewrite.txt
store_id_children 10 startup=5 idle=1 concurrency=0
always_direct allow !getmethod
store_id_access deny connect
store_id_access deny !getmethod
store_id_access allow rewritedoms
reload_into_ims on
max_stale 20 years
minimum_expiry_time 0


refresh_pattern -i squid\.internal 10080 80% 79900 override-lastmod override-expire ignore-reload ignore-no-store ignore-must-revalidate ignore-private ignore-auth

#APPLE STUFF
refresh_pattern -i apple.com/..(cab|exe|msi|msu|msf|asf|wmv|wma|dat|zip|dist)$ 0 80% 43200  refresh-ims

#apple update
refresh_pattern -i (download|adcdownload).apple.com/.*\.(pkg|dmg) 4320 100% 43200 
refresh_pattern -i appldnld\.apple\.com 129600 100% 129600     
refresh_pattern -i phobos\.apple\.com 129600 100% 129600     
refresh_pattern -i iosapps\.itunes\.apple\.com 129600 100% 129600     

# Updates: Windows
refresh_pattern -i microsoft.com/..(cab|exe|msi|msu|msf|asf|wma|dat|zip)$ 4320 80% 43200  refresh-ims
refresh_pattern -i windowsupdate.com/..(cab|exe|msi|msu|msf|asf|wma|wmv)|dat|zip)$ 4320 80% 43200  refresh-ims
refresh_pattern -i windows.com/..(cab|exe|msi|msu|msf|asf|wmv|wma|dat|zip)$ 4320 80% 43200  refresh-ims
refresh_pattern -i microsoft.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 4320 80% 43200 
refresh_pattern -i windowsupdate.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 4320 80% 43200 
refresh_pattern -i windows.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 4320 80% 43200 
refresh_pattern -i .*windowsupdate.com/.*\.(cab|exe) 259200 100% 259200   
refresh_pattern -i .*update.microsoft.com/.*\.(cab|exe|dll|msi|psf) 259200 100% 259200   
refresh_pattern windowsupdate.com/.*\.(cab|exe|dll|msi|psf) 10080 100% 43200 
refresh_pattern download.microsoft.com/.*\.(cab|exe|dll|msi|psf) 10080 100% 43200 
refresh_pattern www.microsoft.com/.*\.(cab|exe|dll|msi|psf) 10080 100% 43200 
refresh_pattern au.download.windowsupdate.com/.*\.(cab|exe|dll|msi|psf) 4320 100% 43200 
refresh_pattern bg.v4.pr.dl.ws.microsoft.com/.*\.(cab|exe|dll|msi|psf) 4320 100% 43200
#windows update NEW UPDATE 0.04
refresh_pattern update.microsoft.com/.*\.(cab|exe) 43200 100% 129600    
refresh_pattern ([^.]+\.)?(download|(windows)?update)\.(microsoft\.)?com/.*\.(cab|exe|msi|msp|psf) 4320 100% 43200  
refresh_pattern update.microsoft.com/.*\.(cab|exe|dll|msi|psf) 10080 100% 43200 
refresh_pattern -i \.update.microsoft.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 525600 100% 525600       
refresh_pattern -i \.windowsupdate.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 525600 100% 525600       
refresh_pattern -i \.download.microsoft.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 525600 100% 525600       
refresh_pattern -i \.ws.microsoft.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 525600 100% 525600       

refresh_pattern ([^.]+\.)?(cs|content[1-9]|hsar|content-origin|client-download).[steampowered|steamcontent].com/.*\.* 43200 100% 43200 reload-into-ims ignore-reload ignore-no-store override-expire override-lastmod
refresh_pattern ([^.]+\.)?.akamai.steamstatic.com/.*\.* 43200 100% 43200 reload-into-ims ignore-reload ignore-no-store override-expire override-lastmod

refresh_pattern -i ([^.]+\.)?.adobe.com/.*\.(zip|exe) 43200 100% 43200 reload-into-ims ignore-reload ignore-no-store override-expire override-lastmod
refresh_pattern -i ([^.]+\.)?.java.com/.*\.(zip|exe) 43200 100% 43200 reload-into-ims ignore-reload ignore-no-store override-expire override-lastmod
refresh_pattern -i ([^.]+\.)?.sun.com/.*\.(zip|exe) 43200 100% 43200 reload-into-ims ignore-reload ignore-no-store override-expire override-lastmod
refresh_pattern -i ([^.]+\.)?.oracle.com/.*\.(zip|exe|tar.gz) 43200 100% 43200 reload-into-ims ignore-reload ignore-no-store override-expire override-lastmod

refresh_pattern -i appldnld\.apple\.com 43200 100% 43200 ignore-reload ignore-no-store override-expire override-lastmod
refresh_pattern -i ([^.]+\.)?apple.com/.*\.(ipa) 43200 100% 43200 ignore-reload ignore-no-store override-expire override-lastmod
 
refresh_pattern -i ([^.]+\.)?.google.com/.*\.(exe|crx) 10080 80% 43200 override-expire override-lastmod ignore-no-cache ignore-reload reload-into-ims ignore-private
refresh_pattern -i ([^.]+\.)?g.static.com/.*\.(exe|crx) 10080 80% 43200 override-expire override-lastmod ignore-no-cache ignore-reload reload-into-ims ignore-private

#FACEBOOK
refresh_pattern ^http?://*.facebook.com/*  10080 80% 43200 override-expire override-lastmod ignore-no-cache ignore-reload reload-into-ims ignore-private

#FACEBOOK IMAGES  
refresh_pattern -i pixel.facebook.com..(jpg|png|gif|ico|css|js)  10080 80% 43200 override-expire override-lastmod ignore-no-cache ignore-reload reload-into-ims ignore-private
refresh_pattern -i .akamaihd.net..(jpg|png|gif|ico|css|js) 10080 80% 43200 override-expire override-lastmod ignore-no-cache ignore-reload reload-into-ims ignore-private   
refresh_pattern -i (facebook.com).(jpg|png|gif) 10080 80% 43200 store-stale override-expire override-lastmod ignore-no-cache ignore-reload reload-into-ims ignore-private 
refresh_pattern static.(xx|ak).fbcdn.net.(jpg|gif|png) 10080 80% 43200 override-expire override-lastmod ignore-no-cache ignore-reload reload-into-ims ignore-private
refresh_pattern (scontent\-lax\d\-\d\.xx|.ak)\.fbcdn.net.*(jpg|gif|png) 10080 80% 43200 override-expire override-lastmod ignore-no-cache ignore-reload reload-into-ims ignore-private

refresh_pattern ^https?://profile.ak.fbcdn.net*.(jpg|gif|png) 10080 80% 43200 override-expire override-lastmod ignore-no-cache ignore-reload reload-into-ims ignore-private

#FACEBOOK VIDEO
refresh_pattern -i .(video-lax\d\-\d\.xx|video\.ak)\.fbcdn.net.*\.(mp4|flv|mp3|amf) 10080 80% 43200 override-expire override-lastmod ignore-no-cache ignore-reload reload-into-ims ignore-private
refresh_pattern (audio|video)/(webm|mp4) 10080 80% 43200 override-expire override-lastmod ignore-no-cache ignore-reload reload-into-ims ignore-private
acl https_login url_regex -i ^https.*(login|Login).*
cache deny https_login

range_offset_limit 512 MB windowsupdate
range_offset_limit 4 MB
range_offset_limit 0
quick_abort_min -1 KB

我使用的是 Squid 包中预先打包的 StoreID 程序，这里是该代码的副本。它位于路径中/usr/local/libexec/squid/storeid_file_rewrite

#!/usr/local/bin/perl

use strict;
use warnings;
use Pod::Usage;

=pod

=head1 NAME

 storeid_file_rewrite - File based Store-ID helper for Squid

=head1 SYNOPSIS

 storeid_file_rewrite filepath

=head1 DESCRIPTION

This program acts as a store_id helper program, rewriting URLs passed
by Squid into storage-ids that can be used to achieve better caching
for websites that use different URLs for the same content.

It takes a text file with two tab separated columns.
Column 1: Regular expression to match against the URL
Column 2: Rewrite rule to generate a Store-ID
Eg:
^http:\/\/[^\.]+\.dl\.sourceforge\.net\/(.*)    http://dl.sourceforge.net.squid.internal/$1

Rewrite rules are matched in the same order as they appear in the rules file.
So for best performance, sort it in order of frequency of occurrence.

This program will automatically detect the existence of a concurrency channel-ID and adjust appropriately.
It may be used with any value 0 or above for the store_id_children concurrency= parameter.

=head1 OPTIONS

The only command line parameter this helper takes is the regex rules file name.

=head1 AUTHOR

This program and documentation was written by I<Alan Mizrahi <[email protected]>>

Based on prior work by I<Eliezer Croitoru <[email protected]>>

=head1 COPYRIGHT

 * Copyright (C) 1996-2023 The Squid Software Foundation and contributors
 *
 * Squid software is distributed under GPLv2+ license and includes
 * contributions from numerous individuals and organizations.
 * Please see the COPYING and CONTRIBUTORS files for details.

 Copyright (C) 2013 Alan Mizrahi <[email protected]>
 Based on code from Eliezer Croitoru <[email protected]>

 This program is free software; you can redistribute it and/or modify
 it under the terms of the GNU General Public License as published by
 the Free Software Foundation; either version 2 of the License, or
 (at your option) any later version.

 This program is distributed in the hope that it will be useful,
 but WITHOUT ANY WARRANTY; without even the implied warranty of
 MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 GNU General Public License for more details.

 You should have received a copy of the GNU General Public License
 along with this program; if not, write to the Free Software
 Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307, USA.

=head1 QUESTIONS

Questions on the usage of this program can be sent to the I<Squid Users mailing list <[email protected]>>

=head1 REPORTING BUGS

Bug reports need to be made in English.
See http://wiki.squid-cache.org/SquidFaq/BugReporting for details of what you need to include with your bug report.

Report bugs or bug fixes using http://bugs.squid-cache.org/

Report serious security bugs to I<Squid Bugs <[email protected]>>

Report ideas for new improvements to the I<Squid Developers mailing list <[email protected]>>

=head1 SEE ALSO

squid (8), GPL (7),

The Squid wiki http://wiki.squid-cache.org/Features/StoreID

The Squid Configuration Manual http://www.squid-cache.org/Doc/config/

=cut

my @rules; # array of [regex, replacement string]

die "Usage: $0 <rewrite-file>\n" unless $#ARGV == 0;

# read config file
open RULES, $ARGV[0] or die "Error opening $ARGV[0]: $!";
while (<RULES>) {
    chomp;
    next if /^\s*#?$/;
    if (/^\s*([^\t]+?)\s*\t+\s*([^\t]+?)\s*$/) {
        push(@rules, [qr/$1/, $2]);
    } else {
        print STDERR "$0: Parse error in $ARGV[0] (line $.)\n";
    }
}
close RULES;

$|=1;
# read urls from squid and do the replacement
URL: while (<STDIN>) {
    chomp;
    last if $_ eq 'quit';

    my $channel = "";
    if (s/^(\d+\s+)//o) {
        $channel = $1;
    }

    foreach my $rule (@rules) {
        if (my @match = /$rule->[0]/) {
            $_ = $rule->[1];

            for (my $i=1; $i<=scalar(@match); $i++) {
                s/\$$i/$match[$i-1]/g;
            }
            print $channel, "OK store-id=$_\n";
            next URL;
        }
    }
    print $channel, "ERR\n";
}

如果这有什么问题，请告诉我。我得到了很多次点击，它工作正常，但是我感觉我在这方面的安全性有些欠缺。我不希望任何容器/监狱/虚拟机能够突破缓存并攻击我的文件系统。我正在使用 Squid 的帮助文件。如果我可以更改任何内容以使其更安全，请告诉我。

参考：

• https://fazar.net/storeid-support-caching-youtube-dan-facebook-https/
• https://github.com/rudiservo/pfsense_storeid
• https://wiki.squid-cache.org/Features/StoreID
• http://wiki.squid-cache.org/Features/StoreID/DB
• https://raw.githubusercontent.com/ucokkarnadi/tempat-sampah/master/squid.conf
• https://github.com/hscbrasil/hsc-dynamic-cache
• https://research.facebook.com/blog/2016/4/the-evolution-of-advanced-caching-in-the-facebook-cdn/