在我提出问题之前,先快速概览一下我正在研究的公司产品:
我们有一台运行 Ubuntu 18.04 和 Vision 应用程序的服务器。该服务器的主要目的是运行 Vision 应用程序。Vision 应用程序将数据记录到 MySQL 数据库(MySQL 安装也基本是出厂设置)。我们还有一个网站,它本质上是数据库的前端。如果 mysql systemd 服务停止运行并且没有自动重新启动,那么 Vision 应用程序将崩溃,并且无法重新启动,直到手动重新启动 MySQL,这是绝对不可接受的(Vision 应用程序必须每天 24 小时运行,停机时间不能超过几秒钟)。
为了保护客户的网络,我们的服务器位于无法访问互联网的隔离本地网络上。
最近发生的事情是 mysql systemd 服务崩溃并且无法重新启动。我发表了这篇文章:
Ubuntu 18.04 服务器上的 mysql.service 位置?
其结果是:
cd /etc/systemd/system
sudo mkdir mysql.service.d
cd mysql.service.d
sudo nano override.conf
然后我在override.conf中输入了以下内容:
[Service]
Restart=always
这并没有解决问题,也就是说,MySQL 在运行 5 到 20 小时后崩溃并且无法重新启动。我检查了 MySQL 崩溃期间的 RAM 和硬盘使用情况,发现两者的使用率都远低于 75%,因此这似乎也不是问题所在。
通过谷歌搜索和浏览,/var/log
我找到了/var/log/unattended-upgrades
。最近一次崩溃发生在 2019 年 1 月 3 日上午 6:28 左右。
cd /var/log/unattended-upgrades
cat unattended-upgrades.log
输出:
(some stuff omitted)
2019-01-02 15:43:33,230 INFO Starting unattended upgrades script
2019-01-02 15:43:33,230 INFO Allowed origins are: o=Ubuntu,a=bionic, o=Ubuntu,a=bionic-security, o=UbuntuESM,a=bionic
2019-01-03 06:28:00,499 INFO Initial blacklisted packages:
2019-01-03 06:28:00,534 INFO Initial whitelisted packages:
2019-01-03 06:28:00,534 INFO Starting unattended upgrades script
2019-01-03 06:28:00,534 INFO Allowed origins are: o=Ubuntu,a=bionic, o=Ubuntu,a=bionic-security, o=UbuntuESM,a=bionic
2019-01-03 06:28:02,248 INFO Removing unused kernel packages: linux-modules-extra-4.15.0-29-generic linux-image-4.15.0-29-generic linux-headers-4.15.0-29 linux-headers-4.15.0-29-generic linux-modules-4.15.0-29-generic
2019-01-03 06:28:36,159 ERROR Auto-removing the packages failed!
2019-01-03 06:28:36,159 ERROR Error message: installArchives() failed
2019-01-03 06:28:36,159 ERROR dpkg returned an error! See /var/log/unattended-upgrades/unattended-upgrades-dpkg.log for details
2019-01-03 06:28:36,561 INFO No packages found that can be upgraded unattended and no pending auto-removals
2019-01-03 07:13:21,059 INFO Initial blacklisted packages:
2019-01-03 07:13:21,137 INFO Initial whitelisted packages:
(some stuff omitted)
下一个命令:
cat unattended-upgrades-shutdown.log
输出:
(some stuff omitted)
Log started: 2019-01-03 06:28:02
(Reading database ... 232457 files and directories currently installed.)
Removing linux-modules-extra-4.15.0-29-generic (4.15.0-29.31) ...
Setting up mysql-server-5.7 (5.7.24-0ubuntu0.18.04.1) ...
Checking if update is needed.
Checking server version.
Error occurred: The mysql.session exists but is not correctly configured. The mysql.session needs SELECT privileges in the performance_schema database and the mysql.db table and also SUPER privileges.
mysql_upgrade failed with exit status 5
dpkg: error processing package mysql-server-5.7 (--configure):
installed mysql-server-5.7 package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
mysql-server-5.7
Log ended: 2019-01-03 06:28:35
我是一个视觉专家,而不是服务器/网络专家,所以我不敢说我完全理解这些日志,但似乎它unattended-upgrades
试图更新 MySQL,但由于没有 root 权限而失败,这完全没有道理,因为unattended-upgrades
几乎所有操作都需要 root 权限。无论如何,经过更多的谷歌搜索,我发现了这一点:
cd /etc/apt/apt.conf.d
cat 20auto-upgrades
输出:
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Unattended-Upgrade "1";
因此,现在我的问题是:
1) 根据以上日志,这是否可能unattended-upgrades
导致 MySQL 服务崩溃?如果是,为什么会这样?
2)由于我们的服务器没有互联网连接,禁用自动更新是否是一个好的做法?
3)还有什么我应该检查的,可能会导致 MySQL 服务崩溃并且不会自动重启?
4) 为什么添加override.conf
withRestart=always
不会使 MySQL 自动重启?当我输入时,sudo systemctl start mysql.service
MySQL 确实成功重启了。这行不是Restart=always
应该自动执行的操作吗?还有其他方法可以输入此行以使其真正起作用吗?
5) 有没有人遇到过unattended-upgrades
导致 MySQL 崩溃的情况?这是一个已知问题吗?我在 Google 上找不到其他人提到这个问题,这似乎很奇怪。
6) 我认为下一步是禁用自动更新。如果是这样,20auto-upgrades
我应该将 设置Update-Package-Lists
为0
、还是 都设置Unattended-Upgrade
为0
?如果无人值守升级已关闭,我认为没有理由让更新软件包列表保持打开状态,因此我倾向于将两者都设置为 0,除非有人另有建议。
答案1
在无人值守升级 mysql 服务器后,我别无选择,只能删除并重新安装 mysql 服务器并恢复数据库。
日志文件样本:
2019-01-24 06:19:16,643 INFO Initial whitelisted packages:
2019-01-24 06:19:16,644 INFO Starting unattended upgrades script
2019-01-24 06:19:16,644 INFO Allowed origins are: o=Ubuntu,a=cosmic, o=Ubuntu,a=cosmic-security, o=UbuntuESM,a=cosmic
2019-01-24 06:19:22,702 INFO Packages that will be upgraded: apt apt-utils libapt-inst2.0 libapt-pkg5.0 mysql-client-5.7 mysql-client-core-5.7 mysql-server mysql-server-5.7 mysql-server-core-5.7
2019-01-24 06:19:22,710 INFO Writing dpkg log to /var/log/unattended-upgrades/unattended-upgrades-dpkg.log
2019-01-24 06:22:03,228 ERROR Installing the upgrades failed!
2019-01-24 06:22:03,242 ERROR error message: installArchives() failed
2019-01-24 06:22:03,256 ERROR dpkg returned a error! See /var/log/unattended-upgrades/unattended-upgrades-dpkg.log for details
Log started: 2019-01-24 06:21:05
Preconfiguring packages ...
(Reading database ... 91129 files and directories currently installed.)
Preparing to unpack .../mysql-server-5.7_5.7.25-0ubuntu0.18.10.2_amd64.deb ...
Unpacking mysql-server-5.7 (5.7.25-0ubuntu0.18.10.2) over (5.7.24-0ubuntu0.18.10.1) ...
Preparing to unpack .../mysql-server-core-5.7_5.7.25-0ubuntu0.18.10.2_amd64.deb ...
Unpacking mysql-server-core-5.7 (5.7.25-0ubuntu0.18.10.2) over (5.7.24-0ubuntu0.18.10.1) ...
Setting up mysql-server-core-5.7 (5.7.25-0ubuntu0.18.10.2) ...
Processing triggers for systemd (239-7ubuntu10.6) ...
Processing triggers for man-db (2.8.4-2) ...
Setting up mysql-server-5.7 (5.7.25-0ubuntu0.18.10.2) ...
Checking if update is needed.
Checking server version.
Running queries to upgrade MySQL server.
mysql_upgrade: (non fatal) [ERROR] 1728: Cannot load from mysql.proc. The table is probably corrupted
mysql_upgrade: (non fatal) [ERROR] 1545: Failed to open mysql.event
mysql_upgrade: [ERROR] 1136: Column count doesn't match value count at row 1
mysql_upgrade failed with exit status 5
dpkg: error processing package mysql-server-5.7 (--configure):
installed mysql-server-5.7 package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
mysql-server-5.7
Log ended: 2019-01-24 06:22:02