![我可以计算出可供公众访问的网站的总大小吗?](https://linux22.com/image/1099371/%E6%88%91%E5%8F%AF%E4%BB%A5%E8%AE%A1%E7%AE%97%E5%87%BA%E5%8F%AF%E4%BE%9B%E5%85%AC%E4%BC%97%E8%AE%BF%E9%97%AE%E7%9A%84%E7%BD%91%E7%AB%99%E7%9A%84%E6%80%BB%E5%A4%A7%E5%B0%8F%E5%90%97%EF%BC%9F.png)
假设我想下载所有公共页面或创建网站 www.psychocats.net 的离线数据库。现在,在开始下载之前,我如何先计算网站的总大小?
答案1
根据类似的问答 -在执行 wget 之前将文件的大小获取到 wget 中?- 我制作了 bash shell 包装器脚本,它可以完全满足您的需要。:)
最新的代码库可以在 Github 上找到:
#!/bin/bash
# Info: https://github.com/mariomaric/website-size#readme
# Prepare wget logfile
log=/tmp/wget-website-size-log
# Do the spider magic
echo "### Crawling ${!#} website... ###"
sleep 2s
echo "### This will take some time to finish, please wait. ###"
wget \
--recursive --level=inf \
--spider --server-response \
--no-directories \
--output-file="$log" "$@"
echo "Finished with crawling!"
sleep 1s
# Check if prepared logfile is used
if [ -f "$log" ]; then
# Calculate and print estimated website size
echo "Estimated size: $(\
grep -e "Content-Length" "$log" | \
awk '{sum+=$2} END {printf("%.0f", sum / 1024 / 1024)}'\
) Mb"
# Delete wget log file
rm "$log"
else
echo "Unable to calculate estimated size."
fi
exit
此外,这个答案也有很大的帮助:Shell 命令对整数求和,每行一个?