我已经苦苦思索了三天了。希望你能告诉我我做错了什么。
我有一个包含 3 个主服务器和 3 个从服务器的 Redis 集群。一个 NodeJS 应用程序希望使用 redis 快速存储和提供数据。一切似乎都很好,但是,我们偶尔会遇到应用程序掉线的情况。结果发现,它经常自行重启。在重启时,应用程序(而不是 redis)会记录以下消息
2020-08-05 08:48 +02:00: events.js:288
2020-08-05 08:48 +02:00: throw er; // Unhandled 'error' event
2020-08-05 08:48 +02:00: ^
2020-08-05 08:48 +02:00:
2020-08-05 08:48 +02:00: Error: connect EMFILE 10.0.0.201:7101 - Local (undefined:undefined)
2020-08-05 08:48 +02:00: at internalConnect (net.js:917:16)
2020-08-05 08:48 +02:00: at defaultTriggerAsyncIdScope (internal/async_hooks.js:311:12)
2020-08-05 08:48 +02:00: at net.js:1007:9
2020-08-05 08:48 +02:00: at processTicksAndRejections (internal/process/task_queues.js:79:11)
2020-08-05 08:48 +02:00: at runNextTicks (internal/process/task_queues.js:66:3)
2020-08-05 08:48 +02:00: at listOnTimeout (internal/timers.js:518:9)
2020-08-05 08:48 +02:00: at processTimers (internal/timers.js:492:7)
2020-08-05 08:48 +02:00: Emitted 'error' event on Socket instance at:
2020-08-05 08:48 +02:00: at emitErrorNT (internal/streams/destroy.js:92:8)
2020-08-05 08:48 +02:00: at emitErrorAndCloseNT (internal/streams/destroy.js:60:3)
2020-08-05 08:48 +02:00: at processTicksAndRejections (internal/process/task_queues.js:84:21)
2020-08-05 08:48 +02:00: at runNextTicks (internal/process/task_queues.js:66:3)
2020-08-05 08:48 +02:00: at listOnTimeout (internal/timers.js:518:9)
2020-08-05 08:48 +02:00: at processTimers (internal/timers.js:492:7) {
2020-08-05 08:48 +02:00: errno: 'EMFILE',
2020-08-05 08:48 +02:00: code: 'EMFILE',
2020-08-05 08:48 +02:00: syscall: 'connect',
2020-08-05 08:48 +02:00: address: '10.0.0.201',
2020-08-05 08:48 +02:00: port: 7101
2020-08-05 08:48 +02:00: }
2020-08-05 08:48 +02:00: events.js:288
2020-08-05 08:48 +02:00: throw er; // Unhandled 'error' event
2020-08-05 08:48 +02:00: ^
2020-08-05 08:48 +02:00:
2020-08-05 08:48 +02:00: Error [ERR_IPC_CHANNEL_CLOSED]: Channel closed
2020-08-05 08:48 +02:00: at ChildProcess.target.send (internal/child_process.js:679:16)
2020-08-05 08:48 +02:00: at Worker.send (internal/cluster/worker.js:45:28)
2020-08-05 08:48 +02:00: at Server.<anonymous> (/totally/secret/folder/structure/app.js:136:16)
2020-08-05 08:48 +02:00: at Server.emit (events.js:311:20)
2020-08-05 08:48 +02:00: at TCP.onconnection (net.js:1554:8)
2020-08-05 08:48 +02:00: Emitted 'error' event on Worker instance at:
2020-08-05 08:48 +02:00: at ChildProcess.<anonymous> (internal/cluster/worker.js:29:12)
2020-08-05 08:48 +02:00: at ChildProcess.emit (events.js:311:20)
2020-08-05 08:48 +02:00: at internal/child_process.js:683:35
2020-08-05 08:48 +02:00: at processTicksAndRejections (internal/process/task_queues.js:79:11) {
2020-08-05 08:48 +02:00: code: 'ERR_IPC_CHANNEL_CLOSED'
2020-08-05 08:48 +02:00: }
2020-08-05 08:48 +02:00: WARNING: NODE_APP_INSTANCE value of '0' did not match any instance config file names.
2020-08-05 08:48 +02:00: WARNING: See https://github.com/lorenwest/node-config/wiki/Strict-Mode
2020-08-05 08:48 +02:00: WARNING: NODE_APP_INSTANCE value of '0' did not match any instance config file names.
2020-08-05 08:48 +02:00: WARNING: See https://github.com/lorenwest/node-config/wiki/Strict-Mode
2020-08-05 08:48 +02:00: WARNING: NODE_APP_INSTANCE value of '0' did not match any instance config file names.
2020-08-05 08:48 +02:00: WARNING: See https://github.com/lorenwest/node-config/wiki/Strict-Mode
2020-08-05 08:48 +02:00: WARNING: NODE_APP_INSTANCE value of '0' did not match any instance config file names.
2020-08-05 08:48 +02:00: WARNING: See https://github.com/lorenwest/node-config/wiki/Strict-Mode
2020-08-05 08:48 +02:00: WARNING: NODE_APP_INSTANCE value of '0' did not match any instance config file names.
2020-08-05 08:48 +02:00: WARNING: See https://github.com/lorenwest/node-config/wiki/Strict-Mode
2020-08-05 08:48 +02:00: (node:28699) UnhandledPromiseRejectionWarning: TypeError: Cannot read property 'emit' of undefined
2020-08-05 08:48 +02:00: at /totally/secret/folder/structure/app.js:323:34
2020-08-05 08:48 +02:00: at runMicrotasks (<anonymous>)
2020-08-05 08:48 +02:00: at processTicksAndRejections (internal/process/task_queues.js:97:5)
2020-08-05 08:48 +02:00: (node:28699) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 530)
2020-08-05 08:48 +02:00: (node:28699) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
2020-08-05 08:48 +02:00: (node:28699) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 201 error listeners added to [Cluster]. Use emitter.setMaxListeners() to increase limit
到目前为止,我尝试禁用 AOF 或 RDB,但没有成功,偶尔的重启仍然会发生。还有什么可能导致此问题?开发人员表示,应用程序根本无法连接到 Redis,因此失败。但这是不可能的。ping 服务器并观察服务的正常运行时间可以 100% 确认,服务器本身始终可访问且可供应用程序使用。全部 6 个。
答案1
错误 EMFILE 是“打开的文件太多”。在打开网络连接的情况下,需要文件描述符,因此由于您要同时打开数千个连接,因此您将遇到打开文件 ulimit,默认情况下为 1024。
您可以通过设置并重新启动 pm2 和您的应用程序来解决此MAX_OPEN_FILES=
问题/etc/default/pm2
。
MAX_OPEN_FILES=8192