服务器的错误日志中出现以下条目。
2022/09/19 23:31:29 [info] 192250#192250: Using 32768KiB of shared memory for nchan in /etc/nginx/nginx.conf:63
2022/09/19 23:31:31 [notice] 192254#192254: signal process started
[ N 2022-09-19 23:31:35.5287 165889/T6 age/Cor/CoreMain.cpp:670 ]: Signal received. Gracefully shutting down... (send signal 2 more time(s) to force shutdown)
[ N 2022-09-19 23:31:35.5288 165889/T1 age/Cor/CoreMain.cpp:1245 ]: Received command to shutdown gracefully. Waiting until all clients have disconnected...
[ N 2022-09-19 23:31:35.5289 165889/Ta Ser/Server.h:901 ]: [ApiServer] Freed 0 spare client objects
[ N 2022-09-19 23:31:35.5289 165889/Ta Ser/Server.h:558 ]: [ApiServer] Shutdown finished
[ N 2022-09-19 23:31:35.5289 165889/T6 Ser/Server.h:901 ]: [ServerThr.1] Freed 0 spare client objects
[ N 2022-09-19 23:31:35.5289 165889/T6 Ser/Server.h:558 ]: [ServerThr.1] Shutdown finished
[ N 2022-09-19 23:31:35.6368 165889/T1 age/Cor/CoreMain.cpp:1325 ]: Passenger core shutdown finished
导致 nginx 服务无法访问，除非手动重启。这种情况每天至少发生一次。
这是出乎意料的，因为没有采取任何行动来启动关闭操作。
更糟糕的是，crontab -e应该每分钟监控故障
*/1 * * * * /opt/launch-crashed-services.sh > /dev/null 2>
脚本未启动或脚本未正确捕获故障。指定的日志为空。
service nginx status | grep 'active (running)' > /dev/null 2>&1
if [ $? != 0 ]
then
sudo service nginx restart > /var/log/nginx/relaunch.log # /dev/null
fi
主要目标是保持 nginx 正常运行：如何监控 cron 作业是否正在触发？
但事实上，nginx 正在接收关闭命令优雅地是关于还有：应该遵循哪些调查途径？
更新
该服务有时会很快崩溃：
Active: failed (Result: core-dump) since Sun 2022-09-25 05:03:16 UTC; 15min ago
Docs: man:nginx(8)
Process: 416872 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
Process: 416873 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
Main PID: 416885 (code=dumped, signal=SEGV)
Tasks: 0 (limit: 2339)
Memory: 68.4M
CGroup: /system.slice/nginx.service
Sep 25 04:50:33 sandbox systemd[1]: Starting A high performance web server and a reverse proxy server...
Sep 25 04:50:33 sandbox systemd[1]: Started A high performance web server and a reverse proxy server.
Sep 25 05:03:16 sandbox systemd[1]: nginx.service: Main process exited, code=dumped, status=11/SEGV
Sep 25 05:03:16 sandbox systemd[1]: nginx.service: Killing process 417241 (nginx) with signal SIGKILL.
Sep 25 05:03:16 sandbox systemd[1]: nginx.service: Killing process 417241 (nginx) with signal SIGKILL.
Sep 25 05:03:16 sandbox systemd[1]: nginx.service: Failed with result 'core-dump'.
以下是输出nginx -T
root@sandbox:~# nginx -T
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
# configuration file /etc/nginx/nginx.conf:
user www-data;
worker_processes auto;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;
events {
worker_connections 768;
# multi_accept on;
}
http {
##
# Basic Settings
##
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
# server_tokens off;
# server_names_hash_bucket_size 64;
# server_name_in_redirect off;
include /etc/nginx/mime.types;
default_type application/octet-stream;
##
# SSL Settings
##
ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3; # Dropping SSLv3, ref: POODLE
ssl_prefer_server_ciphers on;
##
# Logging Settings
##
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
##
# Gzip Settings
##
gzip on;
# gzip_vary on;
# gzip_proxied any;
# gzip_comp_level 6;
# gzip_buffers 16 8k;
# gzip_http_version 1.1;
# gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;
##
# Virtual Host Configs
##
include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;
}
#mail {
# # See sample authentication script at:
# # http://wiki.nginx.org/ImapAuthenticateWithApachePhpScript
#
# # auth_http localhost/auth.php;
# # pop3_capabilities "TOP" "USER";
# # imap_capabilities "IMAP4rev1" "UIDPLUS";
#
# server {
# listen localhost:110;
# protocol pop3;
# proxy on;
# }
#
# server {
# listen localhost:143;
# protocol imap;
# proxy on;
# }
#}
# configuration file /etc/nginx/modules-enabled/10-mod-http-ndk.conf:
load_module modules/ndk_http_module.so;
# configuration file /etc/nginx/modules-enabled/50-mod-http-auth-pam.conf:
load_module modules/ngx_http_auth_pam_module.so;
# configuration file /etc/nginx/modules-enabled/50-mod-http-cache-purge.conf:
load_module modules/ngx_http_cache_purge_module.so;
# configuration file /etc/nginx/modules-enabled/50-mod-http-dav-ext.conf:
load_module modules/ngx_http_dav_ext_module.so;
# configuration file /etc/nginx/modules-enabled/50-mod-http-echo.conf:
load_module modules/ngx_http_echo_module.so;
# configuration file /etc/nginx/modules-enabled/50-mod-http-fancyindex.conf:
load_module modules/ngx_http_fancyindex_module.so;
# configuration file /etc/nginx/modules-enabled/50-mod-http-geoip.conf:
load_module modules/ngx_http_geoip_module.so;
# configuration file /etc/nginx/modules-enabled/50-mod-http-geoip2.conf:
load_module modules/ngx_http_geoip2_module.so;
# configuration file /etc/nginx/modules-enabled/50-mod-http-headers-more-filter.conf:
load_module modules/ngx_http_headers_more_filter_module.so;
# configuration file /etc/nginx/modules-enabled/50-mod-http-image-filter.conf:
load_module modules/ngx_http_image_filter_module.so;
# configuration file /etc/nginx/modules-enabled/50-mod-http-lua.conf:
load_module modules/ngx_http_lua_module.so;
# configuration file /etc/nginx/modules-enabled/50-mod-http-passenger.conf:
load_module /usr/lib/nginx/modules/ngx_http_passenger_module.so;
# configuration file /etc/nginx/modules-enabled/50-mod-http-perl.conf:
load_module modules/ngx_http_perl_module.so;
# configuration file /etc/nginx/modules-enabled/50-mod-http-subs-filter.conf:
load_module modules/ngx_http_subs_filter_module.so;
# configuration file /etc/nginx/modules-enabled/50-mod-http-uploadprogress.conf:
load_module modules/ngx_http_uploadprogress_module.so;
# configuration file /etc/nginx/modules-enabled/50-mod-http-upstream-fair.conf:
load_module modules/ngx_http_upstream_fair_module.so;
# configuration file /etc/nginx/modules-enabled/50-mod-http-xslt-filter.conf:
load_module modules/ngx_http_xslt_filter_module.so;
# configuration file /etc/nginx/modules-enabled/50-mod-mail.conf:
load_module modules/ngx_mail_module.so;
# configuration file /etc/nginx/modules-enabled/50-mod-nchan.conf:
load_module modules/ngx_nchan_module.so;
# configuration file /etc/nginx/modules-enabled/50-mod-stream.conf:
load_module modules/ngx_stream_module.so;
# configuration file /etc/nginx/mime.types:
types {
text/html html htm shtml;
text/css css;
text/xml xml;
image/gif gif;
image/jpeg jpeg jpg;
application/javascript js;
application/atom+xml atom;
application/rss+xml rss;
text/mathml mml;
text/plain txt;
text/vnd.sun.j2me.app-descriptor jad;
text/vnd.wap.wml wml;
text/x-component htc;
image/png png;
image/tiff tif tiff;
image/vnd.wap.wbmp wbmp;
image/x-icon ico;
image/x-jng jng;
image/x-ms-bmp bmp;
image/svg+xml svg svgz;
image/webp webp;
application/font-woff woff;
application/java-archive jar war ear;
application/json json;
application/mac-binhex40 hqx;
application/msword doc;
application/pdf pdf;
application/postscript ps eps ai;
application/rtf rtf;
application/vnd.apple.mpegurl m3u8;
application/vnd.ms-excel xls;
application/vnd.ms-fontobject eot;
application/vnd.ms-powerpoint ppt;
application/vnd.wap.wmlc wmlc;
application/vnd.google-earth.kml+xml kml;
application/vnd.google-earth.kmz kmz;
application/x-7z-compressed 7z;
application/x-cocoa cco;
application/x-java-archive-diff jardiff;
application/x-java-jnlp-file jnlp;
application/x-makeself run;
application/x-perl pl pm;
application/x-pilot prc pdb;
application/x-rar-compressed rar;
application/x-redhat-package-manager rpm;
application/x-sea sea;
application/x-shockwave-flash swf;
application/x-stuffit sit;
application/x-tcl tcl tk;
application/x-x509-ca-cert der pem crt;
application/x-xpinstall xpi;
application/xhtml+xml xhtml;
application/xspf+xml xspf;
application/zip zip;
application/octet-stream bin exe dll;
application/octet-stream deb;
application/octet-stream dmg;
application/octet-stream iso img;
application/octet-stream msi msp msm;
application/vnd.openxmlformats-officedocument.wordprocessingml.document docx;
application/vnd.openxmlformats-officedocument.spreadsheetml.sheet xlsx;
application/vnd.openxmlformats-officedocument.presentationml.presentation pptx;
audio/midi mid midi kar;
audio/mpeg mp3;
audio/ogg ogg;
audio/x-m4a m4a;
audio/x-realaudio ra;
video/3gpp 3gpp 3gp;
video/mp2t ts;
video/mp4 mp4;
video/mpeg mpeg mpg;
video/quicktime mov;
video/webm webm;
video/x-flv flv;
video/x-m4v m4v;
video/x-mng mng;
video/x-ms-asf asx asf;
video/x-ms-wmv wmv;
video/x-msvideo avi;
}
# configuration file /etc/nginx/conf.d/mod-http-passenger.conf:
### Begin automatically installed Phusion Passenger config snippet ###
passenger_root /usr/lib/ruby/vendor_ruby/phusion_passenger/locations.ini;
passenger_ruby /home/jerdvo/.rbenv/shims/ruby;
### End automatically installed Phusion Passenger config snippet ###
# configuration file /etc/nginx/sites-enabled/default:
##
# You should look at the following URL's in order to grasp a solid understanding
# of Nginx configuration files in order to fully unleash the power of Nginx.
# https://www.nginx.com/resources/wiki/start/
# https://www.nginx.com/resources/wiki/start/topics/tutorials/config_pitfalls/
# https://wiki.debian.org/Nginx/DirectoryStructure
#
# In most cases, administrators will remove this file from sites-enabled/ and
# leave it as reference inside of sites-available where it will continue to be
# updated by the nginx packaging team.
#
# This file will automatically load configuration files provided by other
# applications, such as Drupal or Wordpress. These applications will be made
# available underneath a path with that package name, such as /drupal8.
#
# Please see /usr/share/doc/nginx-doc/examples/ for more detailed examples.
##
# Default server configuration
#
server {
listen 80 default_server;
listen [::]:80 default_server;
# SSL configuration
#
# listen 443 ssl default_server;
# listen [::]:443 ssl default_server;
#
# Note: You should disable gzip for SSL traffic.
# See: https://bugs.debian.org/773332
#
# Read up on ssl_ciphers to ensure a secure configuration.
# See: https://bugs.debian.org/765782
#
# Self signed certs generated by the ssl-cert package
# Don't use them in a production server!
#
# include snippets/snakeoil.conf;
root /var/www/html;
# Add index.php to the list if you are using PHP
index index.html index.htm index.nginx-debian.html;
server_name _;
location / {
# First attempt to serve request as file, then
# as directory, then fall back to displaying a 404.
try_files $uri $uri/ =404;
}
# pass PHP scripts to FastCGI server
#
#location ~ \.php$ {
# include snippets/fastcgi-php.conf;
#
# # With php-fpm (or other unix sockets):
# fastcgi_pass unix:/var/run/php/php7.4-fpm.sock;
# # With php-cgi (or other tcp sockets):
# fastcgi_pass 127.0.0.1:9000;
#}
# deny access to .htaccess files, if Apache's document root
# concurs with nginx's one
#
#location ~ /\.ht {
# deny all;
#}
}
# Virtual Host configuration for example.com
#
# You can move that to a different file under sites-available/ and symlink that
# to sites-enabled/ to enable it.
#
#server {
# listen 80;
# listen [::]:80;
#
# server_name example.com;
#
# root /var/www/example.com;
# index index.html;
#
# location / {
# try_files $uri $uri/ =404;
# }
#}
# configuration file /etc/nginx/sites-enabled/fidelity:
server {
server_name sandbox.fdl.club;
root /home/jerdvo/fidelity/current/public;
passenger_enabled on;
passenger_app_env development;
location /cable {
passenger_app_group_name fidelity_websocket;
passenger_force_max_concurrent_requests_per_process 0;
}
# Allow uploads up to 100MB in size
client_max_body_size 100m;
location ~ ^/(assets|packs) {
expires max;
gzip_static on;
}
listen [::]:443 ssl ipv6only=on; # managed by Certbot
listen 443 ssl; # managed by Certbot
ssl_certificate /etc/letsencrypt/live/sandbox.fdl.club/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/sandbox.fdl.club/privkey.pem; # managed by Certbot
include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
}
server {
if ($host = sandbox.fdl.club) {
return 301 https://$host$request_uri;
} # managed by Certbot
listen 80;
listen [::]:80;
server_name sandbox.fdl.club;
return 404; # managed by Certbot
}
# configuration file /etc/letsencrypt/options-ssl-nginx.conf:
# This file contains important security parameters. If you modify this file
# manually, Certbot will be unable to automatically provide future security
# updates. Instead, Certbot will print and log an error message with a path to
# the up-to-date file that you will need to refer to when manually updating
# this file.
ssl_session_cache shared:le_nginx_SSL:10m;
ssl_session_timeout 1440m;
ssl_session_tickets off;
ssl_protocols TLSv1.2 TLSv1.3;
ssl_prefer_server_ciphers off;
ssl_ciphers "ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-SHA";
# configuration file /etc/nginx/sites-enabled/market_sandbox:
server {
server_name provetp.sltfla.online proveat.sltfla.online;
root /home/jerdvo/market/current/public;
passenger_enabled on;
passenger_app_env development;
location /cable {
passenger_app_group_name market_websocket;
passenger_force_max_concurrent_requests_per_process 0;
}
# Allow uploads up to 100MB in size
client_max_body_size 100m;
location ~ ^/(assets|packs) {
expires max;
gzip_static on;
}
listen 443 ssl; # managed by Certbot
ssl_certificate /etc/letsencrypt/live/provetp.sltfla.online/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/provetp.sltfla.online/privkey.pem; # managed by Certbot
include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
}
server {
listen 80;
listen [::]:80;
# server_name prove_tp.sltfla.online prove_at.sltfla.online;
}
server {
if ($host = proveat.sltfla.online) {
return 301 https://$host$request_uri;
} # managed by Certbot
if ($host = provetp.sltfla.online) {
return 301 https://$host$request_uri;
} # managed by Certbot
server_name provetp.sltfla.online proveat.sltfla.online;
listen 80;
return 404; # managed by Certbot
}
# configuration file /etc/nginx/sites-enabled/simon:
server {
server_name simon.domayn.com;
root /home/jerdvo/simon/current/public;
passenger_enabled on;
passenger_app_env development;
location /cable {
passenger_app_group_name simon_websocket;
passenger_force_max_concurrent_requests_per_process 0;
}
# Allow uploads up to 100MB in size
client_max_body_size 100m;
location ~ ^/(assets|packs) {
expires max;
gzip_static on;
}
listen 443 ssl; # managed by Certbot
ssl_certificate /etc/letsencrypt/live/simon.domayn.com/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/simon.domayn.com/privkey.pem; # managed by Certbot
include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
}
server {
if ($host = simon.domayn.com) {
return 301 https://$host$request_uri;
} # managed by Certbot
server_name simon.domayn.com;
listen 80;
return 404; # managed by Certbot
更新 2
systemd.service 有一个正在运行的重启命令。
最后一次重启的时间戳为 [Sept28]，可通过以下代码片段找到dmesg：
[Sep27 23:05] audit: type=1400 audit(1664319928.524:32): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/snap/snapd/17029/usr/lib/snapd/snap-co>
[ +0.000538] audit: type=1400 audit(1664319928.524:33): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/snap/snapd/17029/usr/lib/snapd/snap-co>
[ +0.027718] audit: type=1400 audit(1664319928.552:34): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" n>
[ +0.005538] audit: type=1400 audit(1664319928.556:35): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" n>
[ +0.004074] audit: type=1400 audit(1664319928.560:36): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" n>
[ +0.003893] audit: type=1400 audit(1664319928.564:37): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" n>
[ +0.004513] audit: type=1400 audit(1664319928.568:38): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" n>
[ +0.006500] audit: type=1400 audit(1664319928.576:39): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" n>
[ +0.004354] audit: type=1400 audit(1664319928.580:40): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" n>
[ +0.003309] audit: type=1400 audit(1664319928.584:41): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" n>
[Sep28 00:24] show_signal_msg: 20 callbacks suppressed
[ +0.000008] nginx[56081]: segfault at 10 ip 00007f3350cda593 sp 00007fff52347cc0 error 4 in libperl.so.5.30.0[7f3350c72000+166000]
[ +0.000203] Code: 48 89 43 10 48 83 c4 18 5b 5d 41 5c 41 5d 41 5e 41 5f c3 0f 1f 40 00 0f b6 7f 30 48 c1 e8 03 48 29 f8 48 89 c3 74 89 48 8b 02 <4c> 8b 68 10 4d 85 >
[...]
[Sep30 02:01] nginx[208462]: segfault at 71 ip 00007f7b23ce3593 sp 00007ffede4fee10 error 4 in libperl.so.5.30.0[7f7b23c7b000+166000]
[ +0.000017] Code: 48 89 43 10 48 83 c4 18 5b 5d 41 5c 41 5d 41 5e 41 5f c3 0f 1f 40 00 0f b6 7f 30 48 c1 e8 03 48 29 f8 48 89 c3 74 89 48 8b 02 <4c> 8b 68 10 4d 85 ed 0f 84 28 01 00 00 0f b6 40 30 49 c1 ed 03 49
答案1
您的 nginx 中的 Passenger 模块存在一些问题，这可能是由您网站代码的某些部分触发的。
尝试将 nginx 和 Passenger 升级到最新版本，看看它是否变得更加稳定。